A Software Organization’s Move into Embedded Systems: Embedded Software
Walk into a software organization that’s taking its first steps into embedded systems. You’ll see frustrations over the slow and lackluster development, testing and deployment processes, outcries over outdated tooling, slow speed of development, lack of early feedback, and a general state of awe.
In all respects, embedded software development and embedded systems engineering in general is years, if not decades, behind modern software development. Expecting the HW engineers to readily adopt software processes will inevitably lead to a frustrated blame game. In order to identify areas of improvement, the process needs to be understood, and broken down into basic task types. Let’s first understand the process.
Heterogeneity of Embedded Systems
The hard truth is that whenever a product crosses domain boundaries, the problem complexity explodes and optimization becomes that much harder. Embedded systems engineers, for example, need to think not only about the software but also the hardware running their software. They need to pour through datasheets, look at electrical characteristics, account for component and manufacturing process variations, environmental factors, and much more. Alongside all this, is the embedded software they build. This is not to say that software developers have to deal with a lower complexity, rather to stress the heterogeneity and inflexibility inherent in hardware development.
So what’s the catch, are we doomed to much longer embedded development cycles and seek out the special engineering managers who have seen both worlds and can deal with the heterogeneity?
Learning From The Industry
The answer lies in systematically taking the small wins and improving the key processes of the development organization. Big organizations have done this all in their own flavor and culture and can therefore run the regular timelines. The release cadences of chip manufacturers, phone manufacturers, automotive manufacturers and so on are not only driven by their technical excellence but more so by their process excellence. However, they too have different ways of dealing with issues and what works for Intel will not work for a 1-to-500-person startup but we can drive learnings from their processes, high-level software development practices and work with the cross-domain complexity.
Waterfall vs Agile
Contrary to popular beliefs in the software development community, Waterfall methodology has strong merits in areas where product iteration is costly or conversely, the product is relatively inflexible.
Figure 1: Key decision factor for Waterfall vs Agile
Let’s take the example of an apartment building. The cost of iterating the building structure is immensely high. It could lead to loss of months worth of property value and high material costs. When planning and constructing an apartment building, the designers need to have decades of foresight to make the investment worthwhile. Therefore, in this pure case, the Waterfall methodology makes economic sense.
Establishing that the cost of iteration is the driving factor in our decision, we look to answer the following questions:
- What is the cost of iteration for a certain product?
- Do we desire future flexibility in the product?
- How can we bring the cost of iteration down?
Consider here that embedded software development is a component of the overall product development and should in fact enjoy more flexibility within its realm. We will discuss this in the following.
Tasks Interdependencies Embedded System Development
Let’s identify the types of tasks involved in development of a simple embedded system and their interdependencies.
Figure 2. High-level Dependencies in development of an Embedded System
Figure 2 establishes high-level task dependencies. Development processes are built around these dependencies. It is important to note that the above is NOT the development process. The figure only shows dependencies that constrain the development process, regardless of whether it’s Waterfall or Agile.
The red boxes represent task types which involve hardware. These gravitate inherently towards the Waterfall model with lower release frequency and inflexibility of process. A post will discuss Agile Hardware: How to Bring Down the Iteration Cost of Hardware.
The green color represents embedded software development tasks. This is the code written in, for example C, C++ or Rust, running on an embedded SoC. One would expect this to have the flexibility of modern software development but in many cases, this is not what organizations see in practice. This is due to the dependency on the development platforms as shown.
Key Dependencies
Embedded software development depends heavily on the development platform. The development platform is an accurate representation of the hardware and its environment. An effective development platform can drastically reduce the dependency of embedded software on hardware readiness. This can allow embedded software developers to follow Agile methodologies and have CI/ CD. This can increase productivity not only for the embedded engineers but also for the backend devs, application developers and others who follow agile and interact with the embedded software. The platform may be hardware or software or a combination of both, representing not only the hardware itself but also its environment. In the following, we shall compare various development platform approaches.
1. DevKits – Limited development platform
This approach has hard dependencies for the firmware development on the hardware. The only software before the final hardware is ready is done with dev kits and make-shift hardware. Development kits may be used to create setups which closely resemble the hardware. Upon updates in the hardware, these setups may be modified rather easily. This is only suitable for very low complexity and low volume hardware designs. This approach should be avoided for any serious products.
Pros
- Fast to assemble
- Easy to adapt for hardware changes
- Low cost
Cons
- Not suited for automated HIL testing integration
- Hard to maintain alignment across versions
- Hard to distinguish test equipment faults vs DUT faults
2. Modular Development Platform
A standard development platform is created for the product in which the product is sectioned into functional units which are fast to develop and may be switched out for other ones on design changes.
Pros
- Close representation of the hardware
- Proper versioning and tracking possible
- Well suited for automated HIL testing
Cons
- Relatively higher setup and maintenance cost
3. Software Emulation of the Hardware
If a software emulation of the hardware platform is available, this can be the least costly, easiest to integrate into CI/ CD, and require the least setup effort. It is, however, best suited to digital-only emulation and testing. renode.io is worth checking out in this realm.
Pros
- Easy to set up when emulation models are available
- No physical setup needed, can easily be dockerized and integrated in CI/ CD flows
- Uniform version maintenance with FW since its all software
Cons
- Does not fully represent the final hardware
- Emulation models may not be available for every piece of hardware
- Complex systems include effects beyond what can effectively be represented in a cost effective emulation, e.g. complex mixed signal electronics, high-frequency components.
The following figure shows the relative cumulative development cost and functional coverage of the development platform approaches.
Figure 3. Development platform cost and coverage
This graph assumes the cost over the production, deployment and maintenance of a series hardware. Software emulation and modular hardware development platforms in combination with Hardware-in-the-Loop testing can enable CI/ CD leading to much faster feedback and release cycles and a more robust and tested hardware.
The Way Forward
Looking back at the original problem of the software org stuck with slow Waterfall processes, we can build the new picture:
- The hardware is developed in its Waterfall method with the additional responsibility of building a modular development platform and Hardware-In-the-Loop testing hardware. We will revisit HIL testing and Agile hardware development in a future post.
- Embedded software follows Agile methodologies within their sandbox of the development platform and software emulation.
- The backend, app and other software teams that interface with embedded software can now work closer with embedded software.
