This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Self Updates

In general, the self-update mechanism for operating system level updates is done with two separate partitions. While one partition is the actively booted partition and in use, the other partition can be updated by writing a partition image to it, as it is unused or inactive.

Once the download and writing is done, a reboot is triggered and the boot loader will now switch to the newly updated partition.

If the booting of the updated partition fails, the self update mechanism can revert back to the previous partition or boot to a rescue partition.

Leda Self Update

As updating the running operating system cannot be done at runtime, the approach requires additional disk space, a second partition and also requires the device to be rebooted for the updates to take effect.

In a vehicle, the self-updater cannot decide on its own when to do a reboot, as the vehicle must be in a safe condition (eg parked, state of charge etc.). Hence, the trigger for performaing the switch to another slot and a subsequent reboot is handed over to a higher level component, such as the vehicle update manager, which may in turn rely on driver feedback or other conditions.

Implementation with RAUC Update Service

RAUC is a lightweight update client that runs on your embedded device and reliably controls the procedure of updating your device with a new firmware revision.

For general usage of the RAUC tool, please see the RAUC User manual

Reference configuration

The project contains an example reference implementation and configuration using RAUC, which allows the evaluation of the concepts, mechanisms and involved software components in an emulated, virtual environment.

The Leda quickstart image contains the following disk partitions:

  • a small rescue partition
  • a full SDV installation with a container runtime, pre-loaded SDV container images and deployment specifications and additional developer tools such as nerdctl and kantui.
  • a minimal SDV installation with a container runtime, but no additional examples or developer tools. This partition is used to demonstrate the self-update functionality.
  • additional boot and data partitions for keeping system state information

Note: All three rootfs partitions (rootfs) initially contain the same identical copies of the base operating system. Both SDV Root partitions will use the same shared data partition for the container persistent state.

1 - Self Update Tutorial

This chapter describes the steps necessary to perform a local (without cloud) self update of the operating system.

Self Update Architecture

Self-Update using RAUC Update Bundles

  • On host: Update bundle sdv-rauc-bundle-qemux86-64.raucb is in current folder

    Note: In the development environment, the update RAUC Update Bundle is located in the BitBake machine-specific output folder Example location is tmp/deploy/images/qemux86-64

  • On host: Start a dummy web server for serving the update file

    python3 -m http.server --bind 192.168.7.1
    
  • On host: open two new terminals - one for monitoring and one for triggering the self-update

    • Terminal 1: To view the progress, watch the MQTT topics selfupdate/desiredstate and selfupdate/desiredstatefeedback:

      mosquitto_sub -h 192.168.7.2 -p 1883 -t "selfupdate/#"
      
    • Terminal 2: Trigger the actual self update process by publishing an MQTT message to selfupdate/desiredstate:

      mosquitto_pub -h 192.168.7.2 -p 1883 -t "selfupdate/desiredstate" -f start-update-example.json
      mosquitto_pub -h 192.168.7.2 -p 1883 -t "selfupdate/desiredstate/command" -f download-command.json
      mosquitto_pub -h 192.168.7.2 -p 1883 -t "selfupdate/desiredstate/command" -f update-command.json
      mosquitto_pub -h 192.168.7.2 -p 1883 -t "selfupdate/desiredstate/command" -f activate-command.json
      mosquitto_pub -h 192.168.7.2 -p 1883 -t "selfupdate/desiredstate/command" -f cleanup-command.json
      

      Files:

      start-update-example.json

      download-command.json

      update-command.json

      activate-command.json

      cleanup-command.json

  • Switch to a terminal in the guest

  • On guest: After the self update process completed, check the status:

    rauc status --detailed
    

Self-Update Trigger Message

start-update-example.json file:

{
    "activityId": "random-uuid-as-string",
    "timestamp": 123456789,
    "payload": {
        "domains": [
            {
                "id": "self-update",
                "components": [
                    {
                        "id": "os-image",
                        "version": "${VERSION_ID}",
                        "config": [
                            {
                                "key": "image",
                                "value": "http://leda-bundle-server/sdv-rauc-bundle-minimal-qemux86-64.raucb"
                            }
                        ]
                    }
                ]
            }
        ]
    }
}

Example Message Flows

Current State

  1. Initial response message on startup from self update agent in topic selfupdate/currentstate, or upon request by sending message to selfupdate/currentstate/get

     {
           "activityId": "1234567890",
           "timestamp": 1687510087
     }
    

    Response by Self Update Agent on topic selfupdate/currentstate

     {
           "activityId": "1234567890",
           "timestamp": 1687787461,
           "payload": {
                 "softwareNodes": [
                 {
                       "id": "self-update-agent",
                       "version": "build-152",
                       "name": "OTA NG Self Update Agent",
                       "type": "APPLICATION"
                 },
                 {
                       "id": "self-update:leda-deviceimage",
                       "version": "v0.0.10-194-g3d48cb8",
                       "name": "Official Leda device image",
                       "type": "IMAGE"
                 }
                 ],
                 "hardwareNodes": [],
                 "associations": [
                 {
                       "sourceId": "self-update-agent",
                       "targetId": "self-update:leda-deviceimage"
                 }
                 ]
           }
     }
    

Desired State

  1. External trigger to update via desired state on topic selfupdate/desiredstate:

     {
           "activityId": "<uuid>",
           "timestamp": 123456789,
           "payload": {
                 "domains": [
                       {
                       "id": "self-update",
                       "components": [
                             {
                                   "id": "os-image",
                                   "version": "${VERSION_ID}",
                                   "config": [
                                   {
                                         "key": "image",
                                         "value": "http://leda-bundle-server/sdv-rauc-bundle-qemux86-64.raucb"
                                   }
                                   ]
                             }
                       ]
                       }
                 ]
           }
     }
    

    Response on topic selfupdate/desiredstatefeedback

     {
           "activityId": "1234567890",
           "timestamp": 1687786390,
           "payload": {
                 "status": "IDENTIFYING",
                 "message": "Self-update agent has received new desired state request and is evaluating it.",
                 "actions": []
           }
     }
    
     {
           "activityId": "1234567890",
           "timestamp": 1687786390,
           "payload": {
                 "status": "IDENTIFIED",
                 "message": "Self-update agent is about to perform an OS image update.",
                 "actions": [
                 {
                       "component": {
                             "id": "self-update:os-image",
                             "version": "v0.0.10-194-g3d48cb8"
                       },
                       "status": "IDENTIFIED",
                       "progress": 0,
                       "message": "Self-update agent is about to perform an OS image update."
                 }
                 ]
           }
     }
    
  2. External trigger to download command on topic selfupdate/desiredstate/command:

     {
           "activityId": "1234567890",
           "timestamp": 1687510087,
           "payload": {
                 "baseline": "BASELINE NAME",
                 "command": "DOWNLOAD"
           }
     }
    

    Response on topic selfupdate/desiredstatefeedback

     {
           "activityId": "1234567890",
           "timestamp": 1687786931,
           "payload": {
                 "status": "DOWNLOADING",
                 "message": "Self-update agent is performing an OS image update.",
                 "actions": [
                 {
                       "component": {
                             "id": "self-update:os-image",
                             "version": "v0.0.10-194-g3d48cb8"
                       },
                       "status": "DOWNLOADING",
                       "progress": 0,
                       "message": "Downloading 0.0 MiB..."
                 }
                 ]
           }
     }     
    
    {
          "activityId": "1234567890",
          "timestamp": 1687786936,
          "payload": {
                "status": "DOWNLOAD_SUCCESS",
                "message": "Self-update agent is performing an OS image update.",
                "actions": [
                {
                      "component": {
                            "id": "self-update:os-image",
                            "version": "v0.0.10-194-g3d48cb8"
                      },
                      "status": "DOWNLOAD_SUCCESS",
                      "progress": 100,
                      "message": "Downloaded 106.3 MiB..."
                }
                ]
          }
    }
    
  3. External trigger to update command on topic selfupdate/desiredstate/command:

     {
           "activityId": "1234567890",
           "timestamp": 1687510087,
           "payload": {
                 "baseline": "BASELINE NAME",
                 "command": "UPDATE"
           }
     }
    

    Response on topic selfupdate/desiredstatefeedback

     {
           "activityId": "1234567890",
           "timestamp": 1687787145,
           "payload": {
                 "status": "UPDATING",
                 "message": "Self-update agent is performing an OS image update.",
                 "actions": [
                 {
                       "component": {
                             "id": "self-update:os-image",
                             "version": "v0.0.10-194-g3d48cb8"
                       },
                       "status": "UPDATING",
                       "progress": 0,
                       "message": "Checking bundle version and version in desired state request."
                 }
                 ]
           }
     }
    
    {
          "activityId": "1234567890",
          "timestamp": 1687787228,
          "payload": {
                "status": "UPDATE_SUCCESS",
                "message": "Self-update completed, reboot required.",
                "actions": [
                {
                      "component": {
                            "id": "self-update:os-image",
                            "version": "v0.0.10-194-g3d48cb8"
                      },
                      "status": "UPDATING",
                      "progress": 100,
                      "message": "Writing partition completed, reboot required."
                }
                ]
          }
    }
    
  4. External trigger to activate command on topic selfupdate/desiredstate/command:

     {
           "activityId": "1234567890",
           "timestamp": 1687510087,
           "payload": {
                 "baseline": "BASELINE NAME",
                 "command": "ACTIVATE"
           }
     }
    

    Response on topic selfupdate/desiredstatefeedback

     {
           "activityId": "1234567890",
           "timestamp": 1687787302,
           "payload": {
                 "status": "ACTIVATING",
                 "message": "Self-update agent is performing an OS image activation.",
                 "actions": [
                 {
                       "component": {
                             "id": "self-update:os-image",
                             "version": "v0.0.10-194-g3d48cb8"
                       },
                       "status": "UPDATING",
                       "progress": 0,
                       "message": "Self-update agent is performing an OS image activation."
                 }
                 ]
           }
     }
    
    {
          "activityId": "1234567890",
          "timestamp": 1687787303,
          "payload": {
                "status": "ACTIVATION_SUCCESS",
                "message": "Self-update agent has activated the new OS image.",
                "actions": [
                {
                      "component": {
                            "id": "self-update:os-image",
                            "version": "v0.0.10-194-g3d48cb8"
                      },
                      "status": "UPDATED",
                      "progress": 0,
                      "message": "Self-update agent has activated the new OS image."
                }
                ]
          }
    }
    
  5. External trigger to cleanup command on topic selfupdate/desiredstate/command:

     {
           "activityId": "1234567890",
           "timestamp": 1687510087,
           "payload": {
                 "baseline": "BASELINE NAME",
                 "command": "CLEANUP"
           }
     }
    

    Response on topic selfupdate/desiredstatefeedback

     {
           "activityId": "1234567890",
           "timestamp": 1687787382,
           "payload": {
                 "status": "CLEANUP_SUCCESS",
                 "message": "Self-update agent has cleaned up after itself.",
                 "actions": [
                 {
                       "component": {
                             "id": "self-update:os-image",
                             "version": "v0.0.10-194-g3d48cb8"
                       },
                       "status": "UPDATE_SUCCESS",
                       "progress": 0,
                       "message": "Self-update agent has activated the new OS image."
                 }
                 ]
           }
     }
    
    {
          "activityId": "1234567890",
          "timestamp": 1687787382,
          "payload": {
                "status": "COMPLETE",
                "message": "Self-update completed.",
                "actions": [
                {
                      "component": {
                            "id": "self-update:os-image",
                            "version": "v0.0.10-194-g3d48cb8"
                      },
                      "status": "UPDATE_SUCCESS",
                      "progress": 0,
                      "message": "Self-update agent has activated the new OS image."
                }
                ]
          }
    }
    

Rollback command

  • External trigger to rollback command on topic selfupdate/desiredstate/command:

     {
           "activityId": "1234567890",
           "timestamp": 1687510087,
           "payload": {
                 "baseline": "BASELINE NAME",
                 "command": "ROLLBACK"
           }
     }
    

    Self Update Agent returns in Idle state

Can be used from

  • Downloading (when it is waiting for DOWNLOAD command)
  • Installing (when it is waiting for UPDATE command)
  • Installed (when it is waiting for ACTIVATE command)
  • Failed (when it is waiting for CLEANUP command)

2 - RAUC Integration

Leda integrates RAUC as a reference implementation and example configuration. It allows the evaluation of the concepts, mechanisms and involved software components in an emulated, virtual environment or on physical devices.

Checking the RAUC Status

Get the current RAUC boot status:

rauc status

Example output:

root@qemux86-64:~# rauc status
=== System Info ===
Compatible:  Eclipse Leda qemu86-64
Variant:     
Booted from: rootfs.1 (SDV_B)

=== Bootloader ===
Activated: rootfs.1 (SDV_B)

=== Slot States ===

o [rootfs.1] (/dev/sda5, ext4, inactive)
        bootname: SDV_B
        mounted: /
        boot status: good

x [rootfs.0] (/dev/sda4, ext4, booted)
        bootname: SDV_A
        boot status: good

Forcing to boot the other slot

To manually force the device to boot into another slot, mark the current booted slot as bad, mark the other partitions as active and perform a reboot:

rauc status mark-bad booted
rauc status mark-active other
reboot now

Testing the rescue system

By marking both root slots as bad, the bootloader is supposed to boot the rescue system:

rauc status mark-bad rootfs.0
rauc status mark-bad rootfs.1
reboot now

Example output of rauc:

o [rootfs.1] (/dev/sda5, ext4, inactive)
        bootname: B
        boot status: bad

o [rootfs.0] (/dev/sda4, ext4, booted)
        bootname: A
        mounted: /
        boot status: bad

Customizations

The configurations can be customized by applying or patching the following files:

  • RAUC Configuration file: meta-leda/recipes-bsp/rauc/files/qemux86-64/system.conf
  • Bootloader Configuration file: meta-leda/recipes-bsp/grub/files/grub.cfg
  • The physical disk partition configuration: meta-leda/recipes-sdv/wic/qemux86-grub-efi.wks

RAUC System Configuration

The RAUC System Configuration is the central configuration of the RAUC Update system.

Example:

[system]
compatible=Eclipse Leda qemu86-64
bootloader=grub
grubenv=/grubenv/grubenv
statusfile=/data/rauc.status

[keyring]
path=ca.cert.pem

[slot.efi.0]
device=/dev/sda
type=boot-gpt-switch
region-start=4M
region-size=100M

[slot.rescue.0]
device=/dev/sda3
type=ext4
readonly=true

[slot.rootfs.0]
device=/dev/sda4
type=ext4
bootname=SDV_A

[slot.rootfs.1]
device=/dev/sda5
type=ext4
bootname=SDV_B

GRUB Bootloader Configuration

The GRUB bootloader has a configuration file which describes which partitions are bootable, which partition they are located at and a reference to RAUC’s slot name.

The configuration also contains RAUC specific logic and variables required for a proper integration. Please see the full grub.cfg in the source repository and RAUC Documentation - Integration - GRUB for details.

Excerpt:

...

menuentry "SDV Slot A (OK=$SDV_A_OK TRY=$SDV_A_TRY)" {
    linux (hd0,4)/boot/bzImage root=/dev/vda4 $CMDLINE rauc.slot=SDV_A
}

menuentry "SDV Slot B (OK=$SDV_B_OK TRY=$SDV_B_TRY)" {
    linux (hd0,5)/boot/bzImage root=/dev/vda5 $CMDLINE rauc.slot=SDV_B
}

U-Boot Bootloader Configuration

Similarly to GRUB, integration of RAUC with U-Boot requires custom boot scripting. A highly detailed explaination can, again, be found in the official RAUC Documentation - Integration - U-Boot.

Meta-Leda provides such integration recipes and scripts for all U-boot based targets, for which a Leda Quickstart image is available (qemuarm64, qemuarm and rpi4-64). For example:

Note: A custom U-Boot device defconfig might be required for some devices to be integrated with RAUC. Leda Quickstart images patch the default defconfigs for qemuarm64 and qemuarm to save the U-Boot environment in a VFAT BOOT partition.

Disk Partitioning with OpenEmbedded Image Creator (WIC)

The OpenEmbedded Image Creator is used in BitBake to actually create full disk images with multiple partitions.

These disk images are machine specific and the structure of the partitions are configured in OpenEmbedded Kickstart files (*.wks).

Excerpt qemux86-grub-efi.wks

Note: The excerpt is exemplary, please see the sources for a full representation and documentation.

bootloader --ptable gpt

part --fixed-size 50M --source rawcopy --sourceparams="file=efi-boot.vfat" --fstype=vfat --label boot --active

part --fixed-size 10M --source rawcopy --sourceparams="file=grubenv.vfat" --fstype=vfat --label grubenv

part /rescue --source rootfs --fstype=ext4 --label rescue

part / --source rootfs --fstype=ext4 --label root_a

part / --source rootfs --fstype=ext4 --label root_b

part /data --fixed-size 4G --fstype=ext4 --label data

3 - API Reference

The self update agent (SUA) is a component responsible for the OS Update process SUA is communicating on MQTT interface via usage of defined messages. Internally, SUA uses RAUC to perform the update

Following sequence diagram shows the happy path example of communication between components.

Process Overview

sequenceDiagram participant m as MQTT Broker participant s as SUA participant r as RAUC s -->> m: connect loop Wait for OTA trigger Note left of s: Initial start s ->> m: Current state (installed version from booted partition) Note left of s: Trigger for OTA m ->> s: Desired state request (new version and url to bundle) s ->> m: Feedback (update actions are identified) Note left of s: Command for Download m ->> s: Download command s ->> s: Download bundle s ->> m: Feedback (downloading/downloaded/failed) Note left of s: Command for Update m ->> s: Update command s ->> r: Flash image to partition r ->> r: Flashing... s ->> m: Feedback (updating with percentage) r ->> s: Flash completed/failed s ->> m: Feedback (updated/failed) Note left of s: Command for Activate m ->> s: Activate command s ->> r: Switch partitions (booted <-> other) r ->> s: Switch completed/failed s ->> m: Feedback (activated/failed) Note left of s: Command for Cleanup m ->> s: Cleanup command s ->> s: Remove temporary files s ->> m: Cleanup completed + status from previously failed state
(completed/failed) end
stateDiagram Uninitialized --> Connected: Connected Connected --> Identified: Start (OTA trigger) Identified --> Downloading: Command download Identified --> Failed: If OTA trigger is invalid Downloading --> Updating: Command update Downloading --> Failed: If download has failed Updating --> Activate: Command activate Updating --> Failed: If update has failed Activate --> Cleanup: Command cleanup Activate --> Failed: If activate has failed Failed --> Cleanup: Command cleanup Cleanup --> Idle

Important: Uninitialized state is the default entry state or state in case connection is lost. To simplify reading of the diagram arrows from other states to Unitialized have been removed.

MQTT communication is done over 5 MQTT topics:

Trigger OTA

Topic Direction Description
selfupdate/desiredstate IN This message triggers the update process
The payload shall contain all data necessary to obtain the update bundle and to install it

Trigger self-update step/action

Topic Direction Description
selfupdate/desiredstate/command IN This message triggers the single step in update process (download/flash/activate/cleanup)

Report current state

Topic Direction Description
selfupdate/currentstate OUT This message is being sent either once on SUA start or as an answer to response received by selfupdate/currentstate/get
It contains information about the currently installed OS version

Get current state

Topic Direction Description
selfupdate/currentstate/get IN This message can be received at any point of time
Indicates that SUA should send back the version of the installed OS as current state.

Report status of self-update process

Topic Direction Description
selfupdate/desiredstatefeedback OUT This message is being sent by SUA to share the current progress of the triggered update process
This is the OUT counterpart of selfupdate/desiredstate input message

Checkout

SUA links to some 3rd party libraries, which are fetched as submodules, therefore the cloning shall be performed with recursive option:

git clone --recursive https://github.com/eclipse-leda/leda-contrib-self-update-agent.git

or if was cloned non recursively

git submodule init
git submodule update