Embedded Linux project: Yocto or Ubuntu Core? [Part IV]

Welcome to the concluding chapter of this journey assessing Yocto and Ubuntu Core for your embedded Linux project. 

Among the go-to solutions in the industry and benefiting from wide popularity, Yocto enables developers to build a custom embedded Linux distribution from scratch. With extensive control over every stage of compiling and building the target, Yocto provides extreme flexibility to the expert end-user and kernel engineer. Invariably, it exposes lots of complexity, with developers often facing a steep learning curve before mastering the build process and being able to generate a working image. For an extensive treatment of the pros and cons of Yocto for your embedded Linux project, check out Part I of this series.

Aiming to simplify the journey of embedded Linux development, Ubuntu Core lets enterprises focus on their core business objectives while handling device enablement, board bring-up and maintenance. Head over to Part II to find out whether Ubuntu Core is raising the bar for embedded Linux. Furthermore, Ubuntu Core provides developers what a DIY Linux distro cannot give them: a faster time to market and tested and trusted security and update strategies. In Part III, we reviewed the key considerations behind rolling out your own embedded Linux vs relying on a commercially-supported distribution.

It is now time for an in-depth review of Yocto and Ubuntu Core across a wide range of dimensions: in what follows, we will take a critical stance at the two alternatives in a direct 1:1 comparison. Starting with security and updates, brace for a thorough appraisal of the most widely-adopted Linux solutions for your embedded project. 

Without much further ado, let’s dive straight in.

Security for your embedded Linux project

Security spans a variety of aspects, from mission-critical support and strict confinement to tracking, fixing and timely delivery of security patches.

Some security solutions are available with Yocto. For instance, the meta-security layer aims to harden, protect, and detect vulnerabilities on embedded devices by providing recipes for open-source security packages related to, among others, secure boot, runtime security scanners and intrusion detection. However, when working with Yocto for their embedded Linux project, developers are ultimately responsible for providing and shipping security patches to their images. Also, some Board Support Package (BSP) layers disable security capabilities like cgroup, namespace and BPF support. 

Furthermore, implementing security with manufacturer-provided tools is not easy, as their primary goal is not to deliver good software but rather to sell hardware. For instance,  manufacturers usually optimise a BSP for rapid prototyping, with developers needing further work to make it production-ready. With manufacturers taking open-source software and adding some drivers to it before handing it to their customers, developers are on their own when working on the security of an embedded Linux project. It follows security maintenance often proves a pain point when using Yocto. 

On the other hand, Ubuntu Core takes a security-first approach starting with its architecture design, ensuring built-in security throughout the entire application and device lifecycle. 

The minimal OS, the kernel, and device drivers are packaged and installed as snaps, as are gadget-specific applications. During installation, Ubuntu Core examines the snaps metadata to derive AppArmor profiles, Seccomp filters and device cgroup rules, alongside traditional file permissions (e.g. owner, group, file ACLs). At runtime, Ubuntu Core rigidly sandboxes individual applications via a policy-based system that restricts access to the filesystem, network interfaces, system calls, and other standard Linux facilities. Snaps run confined within a restrictive security sandbox without custom flags at installation. This approach provides strong application confinement and an extraordinary amount of fine-grained security control, guaranteeing adequate protection of the device and any associated data.

Ubuntu Core does not limit its security to snaps. The OS abstracts the root of trust implementation for its secure boot by proving the boot firmware integrity before establishing trust in userspace processes for ARM and x86 SoCs. Furthermore, it supports full disk encryption, protecting both the confidentiality and integrity of a device’s data in case of physical access.

Updates for your embedded Linux project

Traditional Linux software update strategies were non-atomic package-based releases. Whereas an apt-get update works fine for servers within a secure environment, with no power or network outages, intermittent power and network for devices in the field mean an update will likely be interrupted. Delivering reliable software updates to low-powered, inaccessible, and often remotely administered embedded devices requires a well-thought-out solution. 

The target devices of your embedded Linux project will likely connect to the internet, with their attack surface increasing due to the network exposure. Intruders may thus attempt to exploit eventual bugs remotely, and the frequency of the updates will increase due to security issues. 

Traditionally, the components to be updated, i.e. the bootloader, the kernel, the root filesystem, and the applications, differed in the frequency and ease of updates, from being relatively simple but rarely updated to being complicated but most frequently updated. 

Without a secure, robust and fail-safe software update mechanism, your embedded Linux project risks running legacy code and being susceptible to critical exploits. With traditional mechanisms, faulty updates can cause IoT devices to become unstable, and fixing them requires costly manual intervention through on-site engineer visits or device recalls.  

Yocto has no built-in mechanism for over-the-air (OTA) updates, and upgrading software on-Yocto based devices can be tricky. Some third-party, open-source solutions that integrate with Yocto do exist. Among the most popular, Mender.io provides a dual image update scheme designed as an OTA update server and client system written in Go, Python and JavaScript. It supports a full system image update and integrates with Yocto-based images through the meta-mender layer. Available as free and open-source or via paid commercial and enterprise plans, Mender.io requires a fixed layout comprising two A/B partitions with rootfs/kernel, one slot for persistent data and one for U-Boot, and no support for raw NOR, NAND or UBI partitions.

Other popular open-source solutions for updating your embedded Linux project running Yocto are SWUpdate, RAUC and OSTree.

In contrast with Yocto, Ubuntu Core provides out-of-the-box support for OTA delta updates. Following changes to the codebase, snaps will automatically calculate the binary delta to minimize the traffic and time required to distribute the update to your embedded Linux project. 

Snaps have automatic recovery mechanisms such that Ubuntu Core will stop and revert to the previous working version of the application were an error to occur at any point during the update. This means that Ubuntu Core will not install the automated and transactional updates unless they are 100% successful. Also, each update preserves the previous version of code and data, so developers can safely move applications forwards and backwards in their version history. 

Furthermore, the update that ships inside snaps is compressed and remains compressed and read-only through the whole snap lifetime. As noted, Ubuntu Core handles the kernel and base OS as snaps, so all the update benefits relevant to applications also apply to the system foundation.

Time-to-market for your embedded Linux project

Bringing an embedded Linux project to market requires varied capabilities ranging from embedded engineering, app development, backend hosting, software update infrastructure, maintenance and after-sales customer support. Deploying these capabilities at scale typically requires substantial upfront investments, with the complexity involved in acquiring and orchestrating these capabilities delaying the shipment of devices.

Depending on the scope of your embedded Linux project, reducing time to market may be one of your top priorities. With a proliferation of hardware and software solutions rushed out to capture the promise of a multi-billion dollar IoT industry, vendors are under pressure to decrease their development time and speed up their time to market. Selecting a maintainable, scalable, updatable and secure OS from the outset is critical during the development stage to avoid friction once a product is market-ready and shipped in the field.

As Part I argued, the inherent flexibility of Yocto provides developers with control over every stage of compiling and building the target. On the other hand, the extensive configuration options expose significant complexity to the end-users. Recipes in Yocto are used to define builds and are a complex combination of Python, shell scripts, and a BitBake-specific language. Debugging is tricky, and developers often struggle before mastering the build process. One must become familiar with the layers and understand how Bitbake parses before generating a working image with Yocto. Hence, after a particularly steep learning curve and painful debugging sessions during the build process, developers face the harsh reality of not getting to market fast. Competition is catching up or leaving them behind. Developers working on embedded Linux projects built with Yocto have to manage their solution, build their OS, maintain their security and provide updates on their own. To own such a process is rarely the core business objective of an enterprise. Getting to market fast when working on a Yocto-based image is then arguably slow because of its inherent complexity and steep learning curve. 

On the other hand, the combination of Ubuntu Core, its snaps packages and the App Store offers a turn-key, ready-to-go solution that secures developers’ intellectual property. In transitioning from Yocto to Ubuntu Core, the time to market becomes much faster, as engineers that used to spend several months developing the entire stack from the ground up can now focus solely on their value-add software. With an app-centric Ubuntu, developers can focus on building apps as Canonical provides and maintains the low-level components, enabling teams of any size to build highly-secure embedded devices quickly and cost-effectively. 

Furthermore, by providing consulting services allowing enterprises to bootstrap commercial IoT products, Canonical reduces the risks and uncertainty involved in launching and scaling embedded Linux projects while accelerating time to market. The full-service enablement, customisation and development fast tracks an enterprise IoT strategy by guiding it through hardware selection and delivering the infrastructure needed to develop and deploy software to fleets of devices, with consulting services to minimise risk and fill skill gaps. 

Applications for your embedded Linux project

As argued, Ubuntu Core is an app-centric operating system. As the intelligence of a device is ultimately a function of the software it runs, a focus of Ubuntu Core is to make every device effectively app-enabled. The device’s primary function is an app, and developers can then ship other apps next to that primary function. 

According to that vision, Ubuntu Core decouples apps from the OS, acting as an enabling platform running on virtually every hardware on top of which developers may want to put apps.

Furthermore, developers won’t have to build an IoT infrastructure from scratch to roll out their devices. Embedded Linux projects built on Ubuntu Core can get their own hosted and managed IoT App Store, a private application store tailored to software distribution across fleets of devices. It is a custom enterprise store enabling developers to cherry-pick the optimal combination of applications for their devices, including software published in the global Snap Store and custom software developed internally for a specific use case. By building their ecosystem, developers can unlock new business models to monetize their solutions: Ubuntu Core provides OEMs with the ability to commercialize their apps through a private store. 

On the other hand, apps are unconfined and with no strict boundaries on Yocto-based devices. Although there is multi-app support, Yocto does not provide an app store, and there is no direct path toward app monetization as a device manufacturer.

Kernel and OS maintenance for your embedded Linux project

Given the ongoing investment in expensive kernel engineers, manufacturers often opt to go to production with a Yocto-based system but soon realise they are on their own regarding security and upgrades. The maintenance effort and management of variants prove to be unexpected pain points they hadn’t factored in. Manually maintaining a Yocto-based image for an embedded Linux project is arguably a frustrating, ongoing activity distracting enterprises from their core business objectives.  

Also, as different Yocto vendors are not compatible and every image needs to be independently maintained, companies require substantial expertise and skilled kernel engineers in-house for the management of variants and maintenance efforts not to turn into a failed product.

On the other hand, Canonical supports and provides security updates to the base OS, critical software packages and the infrastructure components of Ubuntu Core. The kernel team carefully maintains all Ubuntu kernels and their variants. Via rigorous management of all Linux kernel CVE lists, review and application of all relevant patches for critical kernel defects in the mailing lists, and rigorously testing newly updated kernels end-to-end each SRU cycle, your embedded Linux project is now as secure as your servers. 

Ubuntu Core achieves high kernel reliability through a thoughtful design process, a skilled engineering team, and volume of use in production. While most are familiar with design and engineering, volume and diversity of implementation are perhaps even more noteworthy. The more enterprises and developers rely on it, the more the Ubuntu Linux kernel is rigorously tested, refined, and improved. As most production workloads run on Ubuntu, the Ubuntu kernel is arguably the most production tested kernel in the Linux landscape. 

Board bring-up for your embedded Linux project

Developers need to build the kernel, boot assets and the root file system of their embedded Linux project for board bring-up. Building the codebase requires setting up the boot firmware, loading the kernel with needed drivers, and installing the root file system with the necessary libraries and config files. Indeed, having a bootable board, with all sensors enabled and all peripherals addressable by user-space applications, is critical to boot the target board.

Whereas manufacturers supply the code base for board bring-up as part of the BSPs in Yocto, Canonical enables custom boards to run Ubuntu Core for a fee. 

Kubernetes for your embedded Linux project

Kubernetes established itself as the modern platform to build and run software on the cloud and on-prem. The industry is now shifting its attention to the edge of the compute spectrum.

However, Yocto doesn’t provide a built-in Kubernetes integration. Once more, developers are on their own when looking for an orchestration platform for their embedded Linux project.

On the other hand, Canonical, the publisher of Ubuntu, provides a commercially-supported Kubernetes solution for containerised applications. MicroK8s is a lightweight Kubernetes distribution for clouds, workstations, edges and IoT devices. Combining Ubuntu Core and MicroK8s creates a streamlined, embedded Kubernetes experience optimised for size and performance in IoT and Edge applications.

Running embedded Kubernetes on Ubuntu Core embeds a CNCF-certified orchestration platform into IoT edge solutions with minimal resource consumption and automatic OTA updates. Overall, Ubuntu Core and MicroK8s lead to lower operational costs and provide a robust platform for cloud-native applications.

Further considerations for your embedded Linux project

So far, we discussed security, updates, time-to-market, application ecosystem, kernel maintenance, board bring-up and commercially-supported Kubernetes for your embedded Linux project. Other remaining considerations should inform whether to go with Yocto or Ubuntu Core for your device. 

Among the outstanding aspects to assess is the opportunity cost of allocating resources to embedded Linux development and maintenance vs core business activities.  

Another item worthy of mention is the expected developers’ reach, as an investment in open-source technology stacks and collaboration with a community creates tremendous value for everyone involved. Despite Yocto being very popular professionally and among the preferred go-to platform in the industry, Yocto images differ from one another other. Developers are then forced to target individual devices, preventing scale and global reach.  On the other hand, Ubuntu Core leverages an App store with thousands of apps and developers can ship their code once and immediately reach millions of devices.

Conclusions for your embedded Linux project

Here we are at the end of this mini-series on Yocto vs Ubuntu Core. We tried to provide you with the necessary context to evaluate which of the two approaches to adopt for your embedded Linux project. By now you should have an informed opinion as to whether your use cases warrant proceeding with a DIY or an enterprise-grade, commercially-supported solution.

Further reading for your embedded Linux project

Want to go back to the basics? Find out what is embedded Linux.

Why is Linux the OS of choice for embedded systems? Check out the official guide to Linux for embedded applications in whitepaper or webinar form.

Learn how Ubuntu Core, snaps and Snapcraft create a platform for secure, open-source embedded Linux development and deployment.

Interested in a hearing a detailed comparison of Yocto and Ubuntu Core? Watch the latest webinar on-demand here or download our comparison guide. 

Do you have a question, feedback, or news worth sharing? Join the conversation on IoT Discourse to discuss everything related to the Internet of Things and tightly connected, embedded devices.

Yocto Project and all related marks and logos are trademarks of The Linux Foundation. This presentation is not, in any way, endorsed by the Yocto Project or The Linux Foundation.