I’ve never done either Hardware or Embedded Systems, but a number of my friends have and I’ve always been fascinated by what it would take to extend the reach of Agile to their world. To me it seems the obvious problem is the long cycle times and the cost of getting new hardware fabbed.
There have been several interesting articles and conversations recently that have sparked some digging on my part.
The Players
The people/organizations that I know who seem to be making contributions in this space:
- Nancy Van Schooenderwoert – Agile Rules
- James Grenning – Renaissance Software Consulting
- Johanna Rothmann
- Atomic Objects
Articles
- Embedded Agile: A Case Study In Numbers by Nancy Van Schooenderwoert
- Taming the Embedded Tiger: Agile Test Techniques for Embedded Software (pdf) by Nancy Van Schooenderwoert
- Embedded Extreme Programming Experience Report (pdf) by Nancy Van Schooenderwoert
- PRODUCT DEVELOPMENT AND AGILE METHODS by Jim Highsmith
- What Lifecycle? Selecting the Right Model for Your Project by Johanna Rothman
- Agile Embedded Software Development (Oct 2008) (pdf) by James Grenning
- Test Driven Development for Embedded Software by James Grenning
- Agile Times – Containing Progress Before Hardware – “This issue has many intersting articles to embedded teams and Agile Development topics in general.”
- Atomic Objects Best Practices in Embedded Software
- Danube Case Study with Intel
Quotes and Conversations
My focus has mainly been on the software side of the hardware. One of my early aha moments had to do with freeing the software from target availability, as well as the difficulties in test and debug on hardware. Something I call DOH! Hardware is scarce, not yet developed, expensive, slow to download, etc. Getting the software working independently of the hardware is important to making visible progress without the usual impediments.
For example, client target systems are quite expensive ($1M+), so developers don’t each get one. In that environment you would like to have your code in good working order before running it in the hardware. Of course you won’t find all problems, but instead of finding and fixing all problems in the challenging target environment, the code is well exercised and tested off-target first. Then the kinds of problems found when in the target are integration problems. The target is a good place to look for integration problems and a bad place to look for application problems. I do have one client that has applied some of the agile ideas to hardware development. They have changed their hardware development process so they could do an over-night circuit board build if they needed it.
The key is to have each supplier have a rhythm to their builds/drops to the system integrator. (You have just one supplier? An you’re a supplier to an integrator?) As long as everyone is attempting to build feature-by-feature, and that the hardware people simulate before they go to fab, you can make it.
King Circuit www.kingcircuit.com offers 24 hour turn around on PCBs. Most of the physical constraints we worried about 15 years ago when I was building various embedded DSP solutions are gone. The fast turn cycles of PC motherboards has driven the industry to turn designs into populated hardware quickly.
Great hardware designs are modular. With a bit of thinking H/W engineers can build testable demoable components of the system that can be quickly (days) fabricated. BUT(T) they have to think differently about how they work. That is your challenge, to get them to try it.
There are other solutions too. They have Field Programmable Gate Arrays (FPGAs) that can instantiate a chip design with a download. We loved these things because of the risk they removed and the fact that you could play/test with the design. At the time the # of gates you could get was often smaller than we needed. Now they have huge FPGAs and you can test big designs with them long before taping out to an ASIC. In fact the ASIC guys can just hard code the gate array. Its not as efficient on the gate count but you get a tested design.
One thing you should strive for is getting as much confidence in your software as possible before the hardware arrives. You can reduce potential problems at integration. The kinds of problems at integration are: 1)software does not work, 2)hardware does not work, 3)software and hardware do not talk properly, 4) application does not work.
With TDD and ATDD, along with keeping as much of your code testable independent of the hardware, you should be able to minimize the first and the last problem with this approach. problem 2 can be lessened through programmable devices, prototyping and fast turn hardware builds like Robyn suggests. You may also want to build a automated test or partially automated test that looks out to the hardware. Finding and addressing problem 3 is natural at HW/SW integration time, it goes more smoothly if you are not finding problems you could have found earlier in the software and hardware development cycles.
I’m an embedded software/firmware engineer and my previous employer made ASICS. Before I left there, I explored ways we could adopt some of the agile practices. I’m sure you do chip simulations; those should be done all along instead of waiting for the end, and you probably already know that. Now, testing the software plus the ASIC together as total end-to-end system testing when the chip hasn’t even gone to fab is a different matter, of course. As Robin pointed out, some of the logic could possibly be put on FPGAs or a group of FPGAs. Another approach could be to use a chip simulator which makes a cycle accurate linkable C object, like one we saw from a startup called Carbon Design, but it’s not cheap. And if the system controls mechanicals then you can’t really automate a total end-to-end system test anyway. If not, you’re in luck; most of Carbon Design’s customers seemed to fall into that category, like network fabrics/processors. A more practical approach may be to just handle the ASIC development and software development separately. Have the software employ a good hardware abstraction layer where the chip can be modeled. And be sure to define “unit” when you talk about unit testing, as the term already has a meaning to an ASIC team. Testing a chip “unit” is not the same as “unit testing” in the software sense.
I have been investigating how to use Agile for embedded systems that concurrently develop electo-mechanicals, controllers, and software. So far the best advice I have gotten is to use a product like Simulink from Mathworks as a tool to allow the HW engineers to follow a model driven development approach. SW can be concurrently developed and tested against the simulator.
When the HW design is stable it is sent off from prototypic fab. Then keep the HW simulation current and allow testing of the SW against both the real HW and newer versions of simulated HW.
Sorry there isn’t really any of my own content here – this stuff is more than a bit outside my experience so I have nothing useful to add. Do you have any experience trying to make Agile work for Embedded Software/Hardware? Please share.
Another name to lookup is Ron Morsicato. He worked (or possibly still works) with Nancy at Agile Rules.
Once again, Mark, you are compiling valuable information for the community. Thank you!
We are implementing Scrum as our first agile method. We develop storage devices with software, VHDL and hardware design all involved. So far the software people are starting to grasp the workings and benefits. But the hardware people believe waterfall is the only way and want examples. This collection of articles will go far to get the information we need!
Amazing. Just as I asked about this subject on a couple groups and am looking to help a client in such a situation, this shows up. Serendipity, you make me smile! Thanks so much for this…
Another resource on this topic: “Mocking the Embedded World: Test-Driven Development, Continuous Integration, and Design Patterns” by Michael Karlesky, Greg Williams, William Bereza, Matt Fletcher.
https://www.methodsandtools.com/archive/archive.php?id=59
Just to let you know that you are not alone. As hardware developers we focus on understanding how we can merge the “first time right” methods that we have been perfecting for so long with the lean/agile ideas of early feedback and short iterative cycles. It’s not simple, but we too believe that there is something to be gained – especially in the customer/project experience (which connects heavily with funding :-). A little blurb here: https://www.axcon.dk/blog/process/agile-hardware-development.htm
Feel free to leave comments!
Thanks Mark for a good summary.
On Johanna’s point of good interaction with supplier… one of my clients is struggling with that… They have had a sole source provider of controller boards where the engineers can communicate rapidly during development and therefore they get rapid and accurate protoptyic boards. Unfortunately the procurement department is demanding open bid from multiple sources to ensure lowest price. So the value of the communications channel established over time is not factored in. Anyone else experienced this?
One of my clients are doing agile (now under the hood, since they were taken over by a Six-Sigma heavy-hitter) using set-top-boxes in highly constrained Java environments. They have spent a few years figuring out how to deliver incrementally on an embedded platform, but have begun to get it. Having gone towards minimalism over big-fat-architecture is one area, with highly-modular, decoupled designs which allow for less “just in case” code being deployed when unneeded have all really helped. Their best piece of work shed 90% of the code base, tripled the functionality of the piece, and made 10-100x performance gains over the older module. And this was “arrived at” by test-driving the feature-set from scratch. They retrofitted this system with the old API, and ended up being able to remove the older module whole-sale. It was kind of beautiful, really.
When the projects get more hard(-ware), even into mechanics, we try to blend lean principles with agile. Resulting in a process with milestones/ releases (typically integration gates) because this is important for most of our customers, if you are interested: https://www.solcept.ch/en/company/processes/scprocess.html and https://www.solcept.ch/en/services/engineering/credo.html .
Comments invited…