Agile for Hardware and Embedded Systems

I’ve never done either Hardware or Embedded Systems, but a number of my friends have and I’ve always been fascinated by what it would take to extend the reach of Agile to their world. To me it seems the obvious problem is the long cycle times and the cost of getting new hardware fabbed.

There have been several interesting articles and conversations recently that have sparked some digging on my part.

The Players

The people/organizations that I know who seem to be making contributions in this space:

• Nancy Van Schooenderwoert – Agile Rules
• James Grenning – Renaissance Software Consulting
• Johanna Rothmann
• Atomic Objects

Articles

• Embedded Agile: A Case Study In Numbers by Nancy Van Schooenderwoert
• Taming the Embedded Tiger: Agile Test Techniques for Embedded Software (pdf) by Nancy Van Schooenderwoert
• Embedded Extreme Programming Experience Report (pdf) by Nancy Van Schooenderwoert
• PRODUCT DEVELOPMENT AND AGILE METHODS by Jim Highsmith
• What Lifecycle? Selecting the Right Model for Your Project by Johanna Rothman
• Agile Embedded Software Development (Oct 2008) (pdf) by James Grenning
• Test Driven Development for Embedded Software by James Grenning
• Agile Times – Containing Progress Before Hardware – “This issue has many intersting articles to embedded teams and Agile Development topics in general.”
• Atomic Objects Best Practices in Embedded Software
• Danube Case Study with Intel

Quotes and Conversations

James Grenning:

My focus has mainly been on the software side of the hardware. One of my early aha moments had to do with freeing the software from target availability, as well as the difficulties in test and debug on hardware. Something I call DOH! Hardware is scarce, not yet developed, expensive, slow to download, etc. Getting the software working independently of the hardware is important to making visible progress without the usual impediments.

For example, client target systems are quite expensive ($1M+), so developers don’t each get one. In that environment you would like to have your code in good working order before running it in the hardware. Of course you won’t find all problems, but instead of finding and fixing all problems in the challenging target environment, the code is well exercised and tested off-target first. Then the kinds of problems found when in the target are integration problems. The target is a good place to look for integration problems and a bad place to look for application problems. I do have one client that has applied some of the agile ideas to hardware development. They have changed their hardware development process so they could do an over-night circuit board build if they needed it.

Johanna Rothmann:

The key is to have each supplier have a rhythm to their builds/drops to the system integrator. (You have just one supplier? An you’re a supplier to an integrator?) As long as everyone is attempting to build feature-by-feature, and that the hardware people simulate before they go to fab, you can make it.

Robin Dymond:

King Circuit www.kingcircuit.com offers 24 hour turn around on PCBs. Most of the physical constraints we worried about 15 years ago when I was building various embedded DSP solutions are gone. The fast turn cycles of PC motherboards has driven the industry to turn designs into populated hardware quickly.

Great hardware designs are modular. With a bit of thinking H/W engineers can build testable demoable components of the system that can be quickly (days) fabricated. BUT(T) they have to think differently about how they work. That is your challenge, to get them to try it.

There are other solutions too. They have Field Programmable Gate Arrays (FPGAs) that can instantiate a chip design with a download. We loved these things because of the risk they removed and the fact that you could play/test with the design. At the time the # of gates you could get was often smaller than we needed. Now they have huge FPGAs and you can test big designs with them long before taping out to an ASIC. In fact the ASIC guys can just hard code the gate array. Its not as efficient on the gate count but you get a tested design.

James again:

One thing you should strive for is getting as much confidence in your software as possible before the hardware arrives.  You can reduce potential problems at integration.  The kinds of problems at integration are: 1)software does not work, 2)hardware does not work, 3)software and hardware do not talk properly, 4) application does not work.

With TDD and ATDD, along with keeping as much of your code testable independent of the hardware, you should be able to minimize the first and the last problem with this approach.  problem 2 can be lessened through programmable devices, prototyping and fast turn hardware builds like Robyn suggests.  You may also want to build a automated test or partially automated test that looks out to the hardware.  Finding and addressing problem 3 is natural at HW/SW integration time, it goes more smoothly if you are not finding problems you could have found earlier in the software and hardware development cycles.

Jeanne Petrangelo:

I’m an embedded software/firmware engineer and my previous employer made ASICS. Before I left there, I explored ways we could adopt some of the agile practices. I’m sure you do chip simulations; those should be done all along instead of waiting for the end, and you probably already know that. Now, testing the software plus the ASIC together as total end-to-end system testing when the chip hasn’t even gone to fab is a different matter, of course. As Robin pointed out, some of the logic could possibly be put on FPGAs or a group of FPGAs. Another approach could be to use a chip simulator which makes a cycle accurate linkable C object, like one we saw from a startup called Carbon Design, but it’s not cheap. And if the system controls mechanicals then you can’t really automate a total end-to-end system test anyway. If not, you’re in luck; most of Carbon Design’s customers seemed to fall into that category, like network fabrics/processors. A more practical approach may be to just handle the ASIC development and software development separately. Have the software employ a good hardware abstraction layer where the chip can be modeled. And be sure to define “unit” when you talk about unit testing, as the term already has a meaning to an ASIC team. Testing a chip “unit” is not the same as “unit testing” in the software sense.

John Baker:

I have been investigating how to use Agile for embedded systems that concurrently develop electo-mechanicals, controllers, and software. So far the best advice I have gotten is to use a product like Simulink from Mathworks as a tool to allow the HW engineers to follow a model driven development approach. SW can be concurrently developed and tested against the simulator.

When the HW design is stable it is sent off from prototypic fab. Then keep the HW simulation current and allow testing of the SW against both the real HW and newer versions of simulated HW.

Sorry there isn’t really any of my own content here – this stuff is more than a bit outside my experience so I have nothing useful to add. Do you have any experience trying to make Agile work for Embedded Software/Hardware? Please share.