This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Device Provisioning

1: Manual Provisioning
2: Provisioning with sdv-provision
3: Vehicle Update Manager

3.1: Configuration
3.2: Message Flow

4: Self Updates

4.1: Self Update Tutorial
4.2: RAUC Integration
4.3: API Reference

5: Container Management

5.1: Cloud Connectivity
5.2: Container Update Agent
5.3: Orchestration
5.4: Container Metrics
5.5: Container Registries

The device needs to be configured before it can make a connection to the cloud.

The following initial configuration steps are required:

Create a device in the cloud backend, such as Azure IoT Hub
Configure authentication on device
Configure credentials for accessing private container registries

1 - Manual Provisioning

Follow these steps to do a manual device provisioning:

Generate the device certificate (eg using openssl) and sign it with your CA.
Log in to Azure Portal, Go to Azure Iot Hub and create a new device
Select the proper authentication type, e.g. X.509 Self-signed or X.509 CA Signed
Copy the device certificate (cert file and key file) to the device to /data/var/certificate
Restart cloud connector service or container.

Create a device in Azure IoT Hub

For the device to be connectable, it needs to be known to the cloud service first. In these steps, we will create a new device identity by using Azure IoT Hub.

Pre-Requisites:

Virtual device must already be started with runqemu ... or leda

Note: For Raspberry Pi, please follow the manual steps below and adapt the SSH connection options to the IP of your Raspbery Pi.
The virtual device needs to be remotely accessible via ssh port 2222 on the host’s localhost (Qemu port forwarding in userspace) or via ssh port 22 on the IP address 192.168.7.2 (Qemu virtual networking using TAP network interface)
The container runtime needs to have started successfully, check with sdv-health
A Device has been created in Azure IoT Hub

Note: Do NOT create an “edge” device.

Configure authentication on device

For the proper device authentication, the device management backend authority needs to issue a device-specific certificate and sign it. This is a complex process and subject to the specific situation.

For the Leda quickstart images, the software configuration is prepared with dummy certificates which need to be replaced.

ATTENTION: The Leda example device certificates are public and insecure, they only serve demonstration purposes. You need to replace the intermediate certificates and device certificates with your own.

Generate a device certificate using openssl
Sign it with your intermediate CA certificate
Put it into /data/var/certificate/
Restart the cloud connector service or container: systemctl restart cloud-connector or kanto-cm stop -n cloudconnector --force; kanto-cm start -n cloudconnector

When finished, continue with

Private container registries

Please refer to the Container Registries on how to configure private container registries.

2 - Provisioning with sdv-provision

Meta-leda and the Leda-quickstart image provide a utility sdv-provision that simplifies the device provisioning procedure.

Connecting to the device

Since the procedure includes copying and pasting device IDs, it is recommended to connect to the device over a ssh connection. sdv-motd provides an easy way to find your device’s ip. More information on connecting via ssh can be found here.

Provisioning via a connection string

Device-side

After connecting via ssh run sdv-provision.

Example output:

root@qemux86-64:~# sdv-provision
Checking Eclipse Leda Device Provisioning configuration...
- Certificates directory exists
Checking Device ID
- Based on network device: eth0
- File does not exist, creating: /etc/deviceid
- Device ID: XX-XX-XX-XX-XX-XX
Checking whether either IdScope or ConnectionString is configured
- Neither Id Scope file nor ConnectionString found, needs manual configuration
Do you want to use the global Azure IoT Device Provisioning Service (DPS) by using an Id Scope, or do you want to use a direct connection to a specific Azure IoT Hub using a Connection String?
d) Azure IoT Device Provisioning Service (DPS) with Id Scope
h) Azure IoT Hub with Connection String
Choose:

Note the generated Device ID (XX-XX-XX-XX-XX-XX).
Type h /Azure IoT Hub with Connection String/ and press Enter.
Paste your Azure IoT Hub Connection String: HostName=<IoT Hub in Azure>.azure-devices.net;DeviceId=<XX-XX-XX-XX-XX-XX> and press enter. Where <IoT Hub in Azure> is your Azure IoT Hub name.

Azure Portal

Go to https://portal.azure.com/ and to your Azure IoT Hub named <IoT Hub in Azure>.
Choose Devices -> Add Device.
Enter the Device ID XX-XX-XX-XX-XX-XX generated on the previous step as Device ID.
Pick X.509 Self-Signed and paste the two thumbprints generated by sdv-provision.
Save.

Device-Side

After all of the above steps have been completed, connect back to your device and restart the cloudconnector container by running:

kanto-cm stop -n cloudconnector --force
kanto-cm start -n cloudconnector

Or alternatively use:

$ kantui

And restart the container from the TUI.

3 - Vehicle Update Manager

The Vehicle Update Manager delegates two different types of updates:

The Desired State on the container layer
The Self Update on operating system layer

Vehicle Update Manager Architecture Overview

Desired State

The Desired State is applied at runtime on the container layer.

This type of update mechanism can update vehicle applications, vehicle services and other containers together with configuration resources or data files at runtime. If the applications support it, the rollout can also use high-availability strategies, such as rolling deployments. You can find out more about more about the Container Update Agent here.

Self Update

The Self Update is applied on reboot of the device only.

This type of update mechanism is used for system-level updates which require the operating system to be rebooted to take effect. You can find out more about the Self Update Agent in the tutorial.

3.1 - Configuration

Config file

The default location for the VUM service on the Leda Distro image can be found at /etc/update-manager/config.json. The location of the config file is specified with the --cfg-file [PATH] flag when starting the binary.

Minimal configuration

A minimal configuration (that is used by the Leda Distro) is the following:

{
    "log": {
      "logFile": "/var/log/update-manager/update-manager.log"
    },
    "domain": "vehicle",
    "agents": {
        "containers": {
            "name": "containers",
            "rebootRequired": false,
            "readTimeout": "30s"
        },
        "self-update": {
            "name": "self-update",
            "rebootRequired": true,
            "readTimeout": "30s"
        }
    },
    "thingsEnabled": false
}

Where:

“domain”: specifies the prefix for the MQTT topic, e.g. in this case it’s set to vehicle, so all VUM related topic are prefixed with vehicleupdate/[TOPIC]
“agents”: configures the update agents that would be available to VUM. Here the “readTimeout” key specifies how long should VUM wait for a response from the configured agent.
“thingsEnabled”: whether VUM should use the Kanto Things API for communication or not.

Full configuration

An example configuration file for VUM with all available options is the following one:

{
  "log": {
    "logFile": "log/update-manager.log",
    "logLevel": "ERROR",
    "logFileSize": 3,
    "logFileCount": 6,
    "logFileMaxAge": 29
  },
  "connection": {
    "broker":"www",
    "keepAlive": 500,
    "disconnectTimeout": 500,
    "username":"username",
    "password":"pass",
    "connectTimeout": 500,
    "acknowledgeTimeout": 500,
    "subscribeTimeout": 500,
    "unsubscribeTimeout": 500
  },
  "domain": "vehicle",
  "thingsEnabled": false,
  "rebootEnabled": true,
  "rebootAfter": "1m",
  "reportFeedbackInterval": "2m",
  "currentStateDelay": "1m",
  "phaseTimeout": "2m",
  "agents": {
    "self-update": {
      "rebootRequired": false,
      "readTimeout": "20s"
    },
    "containers": {
      "rebootRequired": true,
      "readTimeout": "30s"
    }
  }
}

Here keys (other the ones above) are self-explanatory. The “connection” section object specifies configuration options for the MQTT connection. Allowed log leves (in order of increasing verbosity)are: ERROR, WARN, INFO, DEBUG, TRACE.

3.2 - Message Flow

Note: This part of the Leda OSS stack is still in active development and there might be differences between the documented and the current version.

As described in Vehicle Update Manager takes a full update message for all domains, identifies the domains affected, the current component versions, actions to be taken, etc., and delegates those actions to the correct update agent (e.g. self-update/container update).

The update manager (UM) in Leda-distro is configured as specified in UM’s config.json, which on the final image is usually located in /etc/update-manager/config.json. The standard two domains supported are containers and self-update, with the latter requiring a reboot on a successful update.

Note: UM allows a custom prefix for all of its topics to be defined (“domain”) in its config.json On the Leda Distro image the default prefix is “vehicle”. If you decide to change it, replace “vehicle” in all MQTT topics mentioned below with your custom prefix.

A full specification of UM’s API and the relevant MQTT topics can be found in its documentation.

Update Message

The general structure of the update message is as follows:

{
  "activityId": "correlation-activity-uuid",
  "timestamp": 123456789,
  "payload": {
    "domains": [
      {
        "id": "domain",
        "config": [],
        "components": [
            {
                "id": "component1",
                "version": "component1-version",
                "config": [
                    {
                        "key": "component1-config-key-1",
                        "value": "component1-config-value-1"
                    }
                ]
            }
        ]
      }
    ]
  }
}

Where multiple domains and components per domain (each with multiple configuration key-value pairs) are allowed. To trigger an update, publish your full update message on the vehicleupdate/desiredstate MQTT topic. When UM receives your message, it splits it across the required domains and publishes messages to the topics that the domain-specific agents are monitoring.

Similarly, update progress can be monitored on the vehicleupdate/desiredstatefeedback topic.

Self-update

The messages for triggering a self-update (image update) are the same as in The Self Update Tutorial. The only difference here is that when you publish a self-update message on the vehicleupdate/desiredstate topic, UM will automatically take care to forward your message to the self-update agent, including forwarding back the update feedback on the vehicleupdate/-namespaced topics.

Containers update (desired state)

Similarly to the self-update, you should start by understanding the operation of the Container Update Agent. After constructing the desired state message you can publish it on the vehicleupdate/desiredstate and UM will, again, forward it to CUA automatically, based on the domain specified in the update message.

Combined update messages

The real strength of UM is deploying updates accross multiple domains with a single update message. For example, if you’d like to deploy an image update bundle (self-update) and a single hello-world container image with the environment variable FOO=BAR set, you can construct the following message:

{
   "activityId":"random-uuid-as-string",
   "timestamp":123456789,
   "payload":{
      "domains":[
         {
            "id":"self-update",
            "components":[
               {
                  "id":"os-image",
                  "version":"${VERSION_ID}",
                  "config":[
                     {
                        "key":"image",
                        "value":"https://leda-bundle-server/sdv-rauc-bundle-minimal-qemux86-64.raucb"
                     }
                  ]
               }
            ]
         },
         {
            "id":"containers",
            "config":[],
            "components":[
               {
                  "id":"hello-world",
                  "version":"latest",
                  "config":[
                     {
                        "key":"image",
                        "value":"docker.io/library/hello-world:latest"
                     },
                     {
                        "key":"env",
                        "value":"FOO=BAR"
                     }
                  ]
               }
            ]
         }
      ]
   }
}

And publish the message on the MQTT topic vehicleupdate/desiredstate. UM will take actions to identify the affected update domains and publish the correct messages on the respective topics. All feedback from the specific update agents will be forwarded back to the UM and published on the vehicleupdate/desiredstatefeedback topic. All these messages will use the same activityId so they can be correlated with each other.

4 - Self Updates

In general, the self-update mechanism for operating system level updates is done with two separate partitions. While one partition is the actively booted partition and in use, the other partition can be updated by writing a partition image to it, as it is unused or inactive.

Once the download and writing is done, a reboot is triggered and the boot loader will now switch to the newly updated partition.

If the booting of the updated partition fails, the self update mechanism can revert back to the previous partition or boot to a rescue partition.

Leda Self Update

As updating the running operating system cannot be done at runtime, the approach requires additional disk space, a second partition and also requires the device to be rebooted for the updates to take effect.

In a vehicle, the self-updater cannot decide on its own when to do a reboot, as the vehicle must be in a safe condition (eg parked, state of charge etc.). Hence, the trigger for performaing the switch to another slot and a subsequent reboot is handed over to a higher level component, such as the vehicle update manager, which may in turn rely on driver feedback or other conditions.

Implementation with RAUC Update Service

RAUC is a lightweight update client that runs on your embedded device and reliably controls the procedure of updating your device with a new firmware revision.

For general usage of the RAUC tool, please see the RAUC User manual

Reference configuration

The project contains an example reference implementation and configuration using RAUC, which allows the evaluation of the concepts, mechanisms and involved software components in an emulated, virtual environment.

The Leda quickstart image contains the following disk partitions:

a small rescue partition
a full SDV installation with a container runtime, pre-loaded SDV container images and deployment specifications and additional developer tools such as nerdctl and kantui.
a minimal SDV installation with a container runtime, but no additional examples or developer tools. This partition is used to demonstrate the self-update functionality.
additional boot and data partitions for keeping system state information

Note: All three rootfs partitions (rootfs) initially contain the same identical copies of the base operating system. Both SDV Root partitions will use the same shared data partition for the container persistent state.

4.1 - Self Update Tutorial

This chapter describes the steps necessary to perform a local (without cloud) self update of the operating system.

Self Update Architecture

Self-Update using RAUC Update Bundles

On host: Update bundle sdv-rauc-bundle-qemux86-64.raucb is in current folder

Note: In the development environment, the update RAUC Update Bundle is located in the BitBake machine-specific output folder Example location is tmp/deploy/images/qemux86-64
On host: Start a dummy web server for serving the update file
```
python3 -m http.server --bind 192.168.7.1
```

On host: open two new terminals - one for monitoring and one for triggering the self-update

Terminal 1: To view the progress, watch the MQTT topics selfupdate/desiredstate and selfupdate/desiredstatefeedback:
```
mosquitto_sub -h 192.168.7.2 -p 1883 -t "selfupdate/#"
```

Terminal 2: Trigger the actual self update process by publishing an MQTT message to selfupdate/desiredstate:

mosquitto_pub -h 192.168.7.2 -p 1883 -t "selfupdate/desiredstate" -f start-update-example.json
mosquitto_pub -h 192.168.7.2 -p 1883 -t "selfupdate/desiredstate/command" -f download-command.json
mosquitto_pub -h 192.168.7.2 -p 1883 -t "selfupdate/desiredstate/command" -f update-command.json
mosquitto_pub -h 192.168.7.2 -p 1883 -t "selfupdate/desiredstate/command" -f activate-command.json
mosquitto_pub -h 192.168.7.2 -p 1883 -t "selfupdate/desiredstate/command" -f cleanup-command.json

Files:

start-update-example.json

download-command.json

update-command.json

activate-command.json

cleanup-command.json

Switch to a terminal in the guest
On guest: After the self update process completed, check the status:
```
rauc status --detailed
```

Self-Update Trigger Message

start-update-example.json file:

{
    "activityId": "random-uuid-as-string",
    "timestamp": 123456789,
    "payload": {
        "domains": [
            {
                "id": "self-update",
                "components": [
                    {
                        "id": "os-image",
                        "version": "${VERSION_ID}",
                        "config": [
                            {
                                "key": "image",
                                "value": "http://leda-bundle-server/sdv-rauc-bundle-minimal-qemux86-64.raucb"
                            }
                        ]
                    }
                ]
            }
        ]
    }
}

Example Message Flows

Current State

Initial response message on startup from self update agent in topic selfupdate/currentstate, or upon request by sending message to selfupdate/currentstate/get

 {
       "activityId": "1234567890",
       "timestamp": 1687510087
 }

Response by Self Update Agent on topic selfupdate/currentstate

 {
       "activityId": "1234567890",
       "timestamp": 1687787461,
       "payload": {
             "softwareNodes": [
             {
                   "id": "self-update-agent",
                   "version": "build-152",
                   "name": "OTA NG Self Update Agent",
                   "type": "APPLICATION"
             },
             {
                   "id": "self-update:leda-deviceimage",
                   "version": "v0.0.10-194-g3d48cb8",
                   "name": "Official Leda device image",
                   "type": "IMAGE"
             }
             ],
             "hardwareNodes": [],
             "associations": [
             {
                   "sourceId": "self-update-agent",
                   "targetId": "self-update:leda-deviceimage"
             }
             ]
       }
 }