This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Container Management

Leda is using Kanto Container Management as the upper-layer container runtime and container orchestration engine.

Besides the command line tool kanto-cm, Kanto also has remote interfaces to manage containers.

Remote Interface

Kanto’s container-management service offers a remote interface via local messaging (MQTT) to interact with a digital twin on the cloud side. This feature can be easily enabled by setting "things": { "enabled": true } in /etc/container-management/config.toml.

When one of the cloud connector components, such as leda-contrib-cloud-connector or Kanto’s azure-connector, is connected to a cloud backend, the container-management will publish its own information using Eclipse Ditto and Eclipse Hono messages. For this, container-management only needs the device Id, gateway Id and tenant Id.

1 - Cloud Connectivity

Simulation of cloud connectivity

  1. Container Management starts and subscribes to edge/thing/response
  2. Cloud Connector starts and publishes the following message to edge/thing/response as soon as the connection is online:
{
    "deviceId":"<namespace>:<gatewayId>:<deviceId>",
    "tenantId":"<tenantId>"
}
  • namespace is azure.edge for Kanto’s Azure Cloud Connector
  • gatewayId indicates the hostname of the Azure IoT Hub
  • deviceId is the identifier for the device, this can either be part of the Azure Connection String or part of the device authentication certificate (CN)
  • tenantId is a configuration setting in the cloud connector

Note: You can simulate the cloud connector trigger by issueing the MQTT message manually on command line:

mosquitto_pub -t 'edge/thing/response' -m '{"deviceId":"dummy-namespace:dummy-gateway:dummy-device-id","tenantId":"dummy-tenant-id"}'

2 - Container Update Agent

About the Container Update Agent (CUA)

The container update agent is a component of kanto container management, that can be enabled in the Container Management’s config.json. When it receives a desired state message on the containersupdate/desiredstate topic it compares the containers (with their current versions and configuration) that are managed by Kanto Container Management and those described in the desired state messages. When it has identified the differences between the two, it resolves them by deleting, downloading, or re-creating containers.

The desired state message

A basic desired state message has a similar structure to all update messages. To deploy the hello-world docker container (removing all other containers currently deployed on the device), you can publish the following desired state message:

{
   "activityId":"test-correlation-activity-uuid",
   "payload":{
      "domains":[
         {
            "id":"containers",
            "config":[],
            "components":[
               {
                  "id":"hello-world",
                  "version":"latest",
                  "config":[
                     {
                        "key":"image",
                        "value":"docker.io/library/hello-world:latest"
                     }
                  ]
               }
            ]
         }
      ]
   }
}

You can configure port mappings, environment variables, device mapping, logging, and more, by adding additional key-value pairs to the “config” array of the given container. For example, if you want to set the environment variable FOO="BAR" inside the container, you can specify:

{
   "activityId":"test-correlation-activity-uuid",
   "payload":{
      "domains":[
         {
            "id":"containers",
            "config":[],
            "components":[
               {
                  "id":"hello-world",
                  "version":"latest",
                  "config":[
                     {
                        "key":"image",
                        "value":"docker.io/library/hello-world:latest"
                     },
                     {
                        "key": "env",
                        "value": "FOO=BAR"
                     }
                  ]
               }
            ]
         }
      ]
   }
}

A full list of all key-value pairs with their defaults can be found in this doc for CUA.

System containers

If a container is not specified in the desired state message, it would be deleted by CUA when the message is received. This might be undesirable for containers that are supposed to run as system services from the first boot of the device through its operation. Such a container might be the Self Update Agent container on the Leda Distro image, which we do not want to specify in every desired state message and deleting/recreating-it during an update might lead to unexpected behaviour. That’s why in Container Management’s config.json in the CUA section, you can specify in the “system_containers” array the names of those containers that would like to be treated as such.

MQTT Topics

Since the MQTT interface of the update agent is abstract, the following message flows are similar to those for the self-update agent mentioned in its tutorial

Getting the current state

On initialization, CUA will publish the current state of all containers on the containersupdate/currentstate topic, the activityId for such a message would be initial-current-state-<TIMESTAMP>. This message can be understood as the “current container inventory”. An example inventory from a running Leda image is currentstate-leda.json.

You can request an update of the current state at any point by publishing an empty message on the: containersupdate/currentstate/get:

{
   "activityId": "<UUID>",
   "timestamp": 123456789,
   "payload": {}
}

The updated inventory will, again, be published on the containersupdate/currentstate topic with correlation ID of <UUID>.

Deploying a desired state message

A desired state message describing the containers and their environments of a typical Leda Distro deployment is leda-desired-state.json. To trigger a desired state deployment directly from CUA, publish the message above on containersupdate/desiredstate, e.g. by running:

   mosquitto_pub -t containersupdate/desiredstate -f leda-desired-state.json

You can monitor the update feedback from CUA, by listening on the topic containersupdate/desiredstatefeedback.

Note: Since the desired state feedback can be quite verbose and happens in real time during the update it could be hard to understand manually. However it’s structured json, so it could be easily machine-parsed and monitored for specific key-value pairs.

3 - Orchestration

Container Management

The container management will respond to the cloud connectivity status message by initially sending a list of containers:

  1. Container Management responds with a list of containers in the topic e/<tenantId>/<gatewayId>:<deviceId>:edge:containers
{
    "topic": "<tenantId>/<gatewayId>:<deviceId>:edge:containers/things/twin/commands/modify",
    "headers": {
        "response-required": false
    },
    "path": "/features/SoftwareUpdatable",
    "value": {
        "definition": [
            "org.eclipse.hawkbit.swupdatable:SoftwareUpdatable:2.0.0"
        ],
        "properties": {
            "status": {
                "softwareModuleType": "oci:container",
                "installedDependencies": {
                    "ghcr.io%2Feclipse%2Fkuksa.val%2Fdatabroker.<uuid>:0.2.5": {
                        "group": "ghcr.io/eclipse/kuksa.val/databroker",
                        "name": "<uuid>",
                        "version": "0.2.5"
                    },
                    "ghcr.io%2Feclipse%2Fkuksa.val.services%2Fseat_service.<uuid>:v0.1.0": {
                        "group": "ghcr.io/eclipse/kuksa.val.services/seat_service",
                        "name": "<uuid>",
                        "version": "v0.1.0"
                    },
                    "ghcr.io%2Feclipse-leda%2Fleda-contrib-cloud-connector%2Fcloudconnector.<uuid>:latest": {
                        "group": "ghcr.io/eclipse-leda/leda-contrib-cloud-connector/cloudconnector",
                        "name": "<uuid>",
                        "version": "latest"
                    },
                    "ghcr.io%2Feclipse-leda%2Fleda-contrib-self-update-agent%2Fself-update-agent.<uuid>:build-20": {
                        "group": "ghcr.io/eclipse-leda/leda-contrib-self-update-agent/self-update-agent",
                        "name": "<uuid>",
                        "version": "build-20"
                    },
                    "ghcr.io%2Feclipse-leda%2Fleda-contrib-vehicle-update-manager%2Fvehicleupdatemanager.<uuid>:latest": {
                        "group": "ghcr.io/eclipse-leda/leda-contrib-vehicle-update-manager/vehicleupdatemanager",
                        "name": "<uuid>",
                        "version": "latest"
                    }
                }
            }
        }
    }
}
  1. Container Management answers with an additional message for each container in the topic e/<tenantId>/<gatewayId>:<deviceId>:edge:containers
{
    "topic": "<tenantId>/<gatewayId>:<deviceId>:edge:containers/things/twin/commands/modify",
    "headers": {
        "response-required": false
    },
    "path": "/features/Container:<uuid>",
    "value": {
        "definition": [
            "com.bosch.iot.suite.edge.containers:Container:1.5.0"
        ],
        "properties": {
            "status": {
                "name": "seatservice-example",
                "imageRef": "ghcr.io/eclipse/kuksa.val.services/seat_service:v0.1.0",
                "config": {
                    "domainName": "seatservice-example-domain",
                    "hostName": "seatservice-example-host",
                    "env": [
                        "VEHICLEDATABROKER_DAPR_APP_ID=databroker",
                        "BROKER_ADDR=databroker-host:30555",
                        "RUST_LOG=info",
                        "vehicle_data_broker=info"
                    ],
                    "restartPolicy": {
                        "type": "UNLESS_STOPPED"
                    },
                    "extraHosts": [
                        "databroker-host:host_ip"
                    ],
                    "portMappings": [
                        {
                            "proto": "tcp",
                            "hostPort": 30051,
                            "hostPortEnd": 30051,
                            "containerPort": 50051,
                            "hostIP": "localhost"
                        }
                    ],
                    "networkMode": "BRIDGE",
                    "log": {
                        "type": "JSON_FILE",
                        "maxFiles": 2,
                        "maxSize": "1M",
                        "mode": "BLOCKING"
                    }
                },
                "createdAt": "2023-02-02T08:46:33.792687313Z",
                "state": {
                    "status": "RUNNING",
                    "pid": 13627,
                    "startedAt": "2023-02-02T09:51:08.049572297Z",
                    "finishedAt": "2023-02-02T09:50:51.752255799Z"
                }
            }
        }
    }
}

4 - Container Metrics

Container Metrics

Container-level metrics, such as CPU utilization or memory usage, can be retrieved on-demand and continuously by enabling the container metrics events. This is done by sending a request to the topic command//<namespaceId>:<gatewayId>:<deviceId>:edge:containers/req/<correlationId>/request:

  1. Enable Container Metrics with a frequency of 5s:
{
    "topic": "dummy-namespace/dummy-gateway:dummy-device-id:edge:containers/things/live/messages/request",
    "headers": {
        "timeout": "10",
        "response-required": true,
        "content-type": "application/json",
        "correlation-id": "3fdc463c-293c-4f39-ab19-24aef7944550"
    },
    "path": "/features/Metrics/inbox/messages/request",
    "value": {
        "frequency": "5s"
    }
}

Example command line:

mosquitto_pub -t command//dummy-namespace:dummy-gateway:dummy-device-id:edge:containers/req/3fdc463c-293c-4f39-ab19-24aef7944550/request -m '{"topic":"dummy-namespace/dummy-gateway:dummy-device-id:edge:containers/things/live/messages/request","headers":{"timeout":"10","response-required":true,"content-type":"application/json","correlation-id":"3fdc463c-293c-4f39-ab19-24aef7944550"},"path":"/features/Metrics/inbox/messages/request","value":{"frequency":"5s"}}'
  1. Container Metrics answers with an additional message for each container in the topic e/<tenantId>/<gatewayId>:<deviceId>:edge:containers
{
    "topic": "<tenantId>/<gatewayId>:<deviceId>:edge:containers/things/twin/commands/modify",
    "headers": {
        "response-required": false
    },
    "path": "/features/Container:<uuid>",
    "value": {
        "definition": [
            "com.bosch.iot.suite.edge.containers:Container:1.5.0"
        ],
        "properties": {
            "status": {
                "name": "seatservice-example",
                "imageRef": "ghcr.io/eclipse/kuksa.val.services/seat_service:v0.1.0",
                "config": {
                    "domainName": "seatservice-example-domain",
                    "hostName": "seatservice-example-host",
                    "env": [
                        "VEHICLEDATABROKER_DAPR_APP_ID=databroker",
                        "BROKER_ADDR=databroker-host:30555",
                        "RUST_LOG=info",
                        "vehicle_data_broker=info"
                    ],
                    "restartPolicy": {
                        "type": "UNLESS_STOPPED"
                    },
                    "extraHosts": [
                        "databroker-host:host_ip"
                    ],
                    "portMappings": [
                        {
                            "proto": "tcp",
                            "hostPort": 30051,
                            "hostPortEnd": 30051,
                            "containerPort": 50051,
                            "hostIP": "localhost"
                        }
                    ],
                    "networkMode": "BRIDGE",
                    "log": {
                        "type": "JSON_FILE",
                        "maxFiles": 2,
                        "maxSize": "1M",
                        "mode": "BLOCKING"
                    }
                },
                "createdAt": "2023-02-02T08:46:33.792687313Z",
                "state": {
                    "status": "RUNNING",
                    "pid": 13627,
                    "startedAt": "2023-02-02T09:51:08.049572297Z",
                    "finishedAt": "2023-02-02T09:50:51.752255799Z"
                }
            }
        }
    }
}

  1. To disable Container Metrics, send a request with frequency of 0s:

    mosquitto_pub -t command//dummy-namespace:dummy-gateway:dummy-device-id:edge:containers/req/3fdc463c-293c-4f39-ab19-24aef7944550/request -m '{"topic":"dummy-namespace/dummy-gateway:dummy-device-id:edge:containers/things/live/messages/request","headers":{"timeout":"10","response-required":true,"content-type":"application/json","correlation-id":"3fdc463c-293c-4f39-ab19-24aef7944550"},"path":"/features/Metrics/inbox/messages/request","value":{"frequency":"0s"}}'

5 - Container Registries

When deploying containerized applications, the container runtime will pull container images from a (remote) container registry.

The pulled container images and their layers are then stored in a local storage.

Private Container Registries

To be able to pull container images, the container runtime needs access to the container registry. Some container registries require authentication. The Kanto Container Manager can be configured to use credentials when accessing remote container registries.

In the Leda images, the sdv-kanto-ctl tools allows to easily add authentication to the container manager configuration:

sdv-kanto-ctl add-registry -h <registryhostname> -u <your_username> -p <your_password>

For example, to access container images from GitHub Packages in a private repository, you need a GitHub Personal Access Token (PAT) with the read: packages scope. Then, add the repository as shown below:

sdv-kanto-ctl add-registry -h ghcr.io -u github -p <Your_GitHub_PersonalAccessToken>

sdv-kanto-ctl will make the necessary modifications to /etc/container-management/config.json and restarts the container-management.service systemd unit, so that the changes take effect. You may need to recreate or restart the container if a previous pull failed.

Please see the Eclipse Kanto Container Manager Configuration reference for details.