Nate Domin | Software Engineer

Simulate Hardware With Tests

I’ve long been fascinated by the idea of automated testing of production software. The problem I faced in implementing anything like “automated testing” was that I couldn’t figure out how to do it with embedded firmware.

Writing firmware can be tricky. There are often dependencies that you can’t control. Typical problems include: How do you test complicated protocol stacks? How can you test the control of analog hardware whose behavior changes with temperature and/or humidity? How can you automate observations of whether the LED is blinking?

In my many years of working as a “seasoned” embedded software engineer, I have been responsible for the creation of multiple large gobs of code that have seemed difficult or even impossible to test.

My VHDL-slinging coworkers would often share their experiences of creating “test benches” to verify their complicated FPGA designs. It sounded good but these same coworkers would end up walking away in frustration when I started asking questions like: “Yea, but how can I write a test case for this piece of hardware that has dramatically different analog responses over time?”

Some of you might be more evolved than I was and may have just rattled off a bunch of answers to the questions above. My problem was I just didn’t understand how to do it. I knew in my heart that there had to be a way but I didn’t understand what it was.

I often looked in envy at my peers that were writing server-based and PC-based code. I’d mockingly say to myself: “It must be nice to work with ‘testable software’.” – I would talk myself into believing it was impossible to develop meaningful tests for embedded firmware.

At one really low point of my career, I even went as far as to learn VHDL in the hopes that working with test benches would help show me a path forward. – I was beginning to understand what my coworkers had been talking about but I still failed to see how it could be applied in a meaningful way to the art of embedded software.

Then I discovered James Grenning‘s book, Test Driven Development for Embedded C. It took me a while to work through his examples but it felt as though this book was specifically written to address my concerns, systematically. – I’ve since discovered others writing about the topic, which is awesome, but this was the first light of hope that I had ever seen.

To help myself work thorough my frustrations/confusion, I took a software module I had written ages ago and used it to create some test cases that could be called any time it was compiled with GCC on Linux (long live make!).

It was all interesting but I needed a project to apply the knowledge to.

It took a while to identify a good pilot project but it eventually came in the form of a need to develop a firmware driver that could dynamically discover the midpoint of a noisy sinusoidal curve. (Sounds like a crazy piece of hardware, right?) I had finally found an excuse to develop a piece of production embedded code using TDD!

My tools were a little clunky since I was choosing to use a Linux Virtual Machine to run my test suite on but it worked! I was able to develop a working solution relatively quickly, with a significantly shorter code-debug-test-repeat development cycle.

There were a few challenges along the way:

It was tough to effectively sync code changes between the development machine and the test-target, a Linux Virtual Machine. Distributed Version Control (long live Mercurial!) quickly solved that problem.
Collecting reasonable data to model tests with. This was solved thanks to a very patient test engineer and a bit of Python code to interact with “dumb” firmware that could perform analog reads-and-writes.
But! The. Biggest. Problem. I fought was during the integration of the firmware with the real target hardware. There was a behavior in the real system that was never effectively captured by my model-generating firmware and Python code. The target hardware had some hysteresis effects that I failed to capture in my models due to the slower read-write cycle of my PC-based controller. This made for a very frustrating integration ~~session~~ month but in the end, we were successful. I had been working with an inaccurate model and it took a frustrating amount of time to recognize the hysteresis.

The takeaway is that TDD for embedded projects is far more feasible than I once thought. In talking with some of my colleagues that also write embedded software, it seems a bit atypical in practice but I believe it should be a best-practice for our field and should probably be embraced. I certainly need to refine my processes but there are a lot of benefits to this style of development.

Flawed Conclusions

I once heard about a researcher that was studying a rare breed of frog that had some incredible jumping abilities. On a good day, with wind at its back, it was said the species could jump a good 30+ feet. This may not sound impressive to you but… for a frog? Trust me, it’s impressive.

Anyways, this researcher secured funding and managed to find a frog to study. He was confident he could uncover the frog’s jumping secrets.

On a rainy Monday, he put the frog down on the floor and yelled, “Jump Frog Jump!” The frog jumped. After logging many jumps, the he was elated to record in his notebook that the frog had jumped an average of 27.2 feet.

Then, on Tuesday, the researcher cut off one of the frog’s front legs.

I suspect you are now thinking to yourself, “Wait. He cut off one of the frog’s legs? What value is there in a story about a three-legged frog?” Please be assured that there’s a point in here somewhere. Stay with me.

So… on that Tuesday, the researcher went ahead with his experiments. He asked the three-legged frog to “Jump Frog Jump!” The frog jumped. After many jumps, he recorded that the frog had jumped an average of 23.5 feet. Very interesting.

Wednesday went similarly. The researcher cut off the second of the frog’s front legs and coaxed the two-legged frog to “Jump Frog Jump!” The frog jumped an average of 21.2 feet. Wow. This was some frog!

On Thursday, the researcher cut off one of the back legs and encouraged the one-legged frog to “Jump Frog Jump!” The frog jumped 7 feet!

Spurred by his success, the researcher removed the frog’s last leg on Friday and yelled, “Jump Frog Jump!”

The leg-less frog didn’t jump. It just sat there as if nothing had happened. The researcher tried yelling at the frog again. “JUMP Frog JUMP!”

Still nothing. The researcher was getting frustrated and yelled as loud as he could, “JUMP! FROG! JUMP!” This went on for an uncomfortable amount of time without any jumps.

Finally, disheartened, the researcher recorded his findings:

“Frogs without legs don’t listen.”

In life and in work, it is sometimes important to step back from a situation to figure out if we are asking the wrong questions when we get results that we don’t want or expect.

Are our expectations wrong? Does the team understand what’s wrong with the product? Does the customer know what they want? Have we gone about solving the problem in an incorrect way? Do we even understand the problem to begin with?

As professionals, we need to constantly manage assumptions, work to understand the problems we are solving, and work to communicate vision. Otherwise, we’re just yelling at a legless frog. And that’s sad.

No frogs were harmed in the writing of this post.

Polishing the Curves

Tell me if this has ever happened to you.

When going over some code, I discovered a chunk of “working” code that had an incredibly … amateurish implementation. The sad part wasn’t that the code-quality was poor. No, the sad part was: I was the fool that had written it.

In my defense, these perceived quality-issues were because I didn’t know any better at the time of implementation. Back when I had written the module, I had just started my efforts to learn a new-to-me framework, wxPython. And to further my defense, that code has now been working for years without any bug fixes needed. Shouldn’t it’s longevity be an endorsement of its implicit quality?

I argue the answer is yes! But that is not the point of this post.

I was reminded of this code-discovery while watching this video, featuring Ira Glass. With amazing elegance, Ira discusses creativity and the inevitable frustration we beginners face in our attempts to achieve results that match our “taste”. It’s a good video. You should watch it. It’s a belief that seems to echo Malcolm Gladwell’s “10,000-Hour Rule“.

I don’t know about you but I find it reassuring to realize that frustrations with early work is a nearly universal problem.

The twist I’ve discovered is that even “experienced” software engineers like myself fight these frustrations more frequently than one might expect. This can happen in a variety of cases, including: starting new projects in new domains, learning new programming frameworks, and sometimes even when working with new people (e.g. teammates, customers, Sith Lords).

I don’t think I am alone. Unless we stop trying to grow, we humans are almost always engaged in battles with learning curves. It’s important to recognize them.

As Ira teaches, we have to ignore our inevitable frustrations and keep producing as we struggle to conquer these various learning curves. Producing and polishing our work is the only way to improve our skill enough to match our expectations.

And then! We need to share our poor early attempts with the world so others can gain discover hope and have a good laugh.

Version Control is a Solved Problem

There are a lot of really interesting open problems in software but there’s one problem that I keep seeing professionals walk into that deserves a mention. Not because it’s an unsolved problem but because it has actually been solved. Many. Times. Repeatedly.

Let me cut to the chase. If you write software, you need to use version control. If your team has more than zero developers developing code, then you need to be using version control. If you write scripts, or C, or VHDL, or MATLAB, or IDL, or R, or HTML, or Python, or CSS or … really anything that is captured in a text editor, you really need to be using version control.

For many, this may not sound like a big revelation. Version Control is one of the “solved problems” of our day. But it may amaze you to discover that there are lot of people out there who are failing to use these tools.

Over the last few years, I seem to have run across a surprising number of fringe groups that have held on to the notion that they do not need to use formal version control tools. They’re the “Senior Scientists”, the “Test Engineers”, the “Software Prototypers”, the “Electrical Engineers”, the “System Administrators”, the “Meteorologists”, and the “hobbyists” that have often accidentally stumbled into the world of software development or think that their little application is just a proof-of-concept.

Please don’t confuse my soap-box-preaching as a condemnation of the amazing work that most of these people do. They are heroes for figuring out smart new ways to solve their problems. Writing software is not and should not be an elitist activity left only to the cubicle-dwelling software engineers. Everyone should be learning how to get computers to do their bidding. The point I am trying to make is that hacking away at a problem is only getting you to first base when you could instead be getting a home run.

The good news is that Version Control can be FUN and once you learn how to use it effectively, your software-centric life is going to improve in countless ways. Some of the benefits include:

Inherent backups of previous work, allowing retrieval of “dead code”
Supports seeing what has changed between versions.
Facilitates collaboration with others.
- Syncing code changes among other developers (bug fixes, new features)
- Allows for meta conversations about the codebase (what is changing?)
Supports Finding/Fixing Bugs in a repeatable fashion
Provides an “Undo Button” to go back to previous checkpoints

Some accidental software engineers eventually adopt “manual version control systems” that revolve around date-stamped copies of directories. While this is a generally poor way to do it, it is better than nothing.

If you are a person writing software in the “shadows” and failing to use version control, please do not be ashamed. — I am just asking that you strongly consider adopting a tool and learning how to use it. If you’re looking for a relatively easy place to start, take a look at Mercurial. There’s a really great introduction available at http://hginit.com.

Version control is one of the solved problems of our universe. There are some really great tools available and a surprising number of them are free (examples: Mercurial, Git, Subversion) with great tutorials available to learn and use them. It is definitely worth investing the time to improve your workflow and learn how to use a formal Version Control System.

If your development team has more than zero people on it, you need version control. Please explore the opportunities.

Optimize Second

When I was first starting out as an engineer, I was tasked with a job that sounded easy but quickly became a challenge for me.

I was working with a senior engineer to develop a embedded controller for a piece of proprietary hardware and our system requirements called for frequent polynomial fits based on raw sensor data, at runtime, in the firmware.

I know what you’re thinking. This sounds like a pretty easy task, right?

Now, in most software contexts, it’s safe to assume that there will be a prewritten software library that can be called by your application to perform a “simple” least-squares polynomial fit. If you’re writing your application for a PC, you probably won’t have any problems using Google to discover the basis for a good solution (Python has a great polyfit routine built into numpy). Actually, thanks to Github, you’ll can find pretty good solutions in nearly. any. language. pretty quickly.

Unfortunately, back in the year 2000, it was frustratingly difficult to find a starting point, let alone a working polynomial fit routine. To make matters worse, our target platform was an 8-bit 8051 microcontroller that was severely underpowered for the complete application.

I spent the better part of a month struggling to get solutions from the
Numerical Recipes book to work in addition to fruitless searches in libraries and bookstores.

When I finally found an algorithm that worked with my C-compiler, I was elated! The algorithm worked consistently but had an unfortunate side effect. It required four times longer to complete than my allotted execution time.

I wasn’t worried though. Having a working piece of code is the first step in succeeding with software. Incremental refinement of a viable solution is what makes software development fun.

The hard part is to first identify a process that works. Refining that process to achieve a targeted result with the known constraints requires effort but often isn’t nearly as difficult as identifying a workable solution first.

Often, we engineers get so caught up in trying to solve a problem “efficiently” that we immediately discount potentially great solutions that meet or exceed the requirements and are inherently more maintainable. This is almost always a mistake.

If you are choosing to implement something in assembly because its resultant executables are fast and you think you need a fast solution, that’s fine… but until you see how slow the ~~[insert high level, more maintainable choice here]~~ Python version is, how do you know that you really need to spend the extra development time to write and debug assembly code?

I believe it is always smarter to start with a “simple”, “workable” solution first and iterate it to meet all of the requirements. Optimization should always be “Step Two.”

Ideas Are The Easy Part

A friend of mine has a really neat idea. He wants to develop an online tailoring service that will leverage skilled people overseas to produce suits and clothing for people here in the U.S.A. In case you’ve never experienced custom made clothing, the fit is truly amazing and will almost always leave you wanting more.

Sure, there are a few already-existing companies that do this but my friend thinks there are a few “flaws” in their business models that could be tweaked to result in a successful business.

It’s a neat idea. Ideas are great. I love ideas. The problem with ideas is that it is often convenient to forget that ideas can require a lot of ~~effort~~ work to bring them to fruition.

Coming up with the idea behind this business was the easy part. Anyone could do that. The difficult part is in turning the idea into a reality. Potential implementation issues I can imagine include: fashion trends are notoriously fickle, supply chain management across continents, importing from overseas is a logistical challenge, good website design takes time, and… oh yea, that little thing called marketing.

There’s an old joke in the software engineering world about pigs and chickens that goes something like this:

A chicken and a pig decide to open a restaurant that serves breakfast. The chicken takes some initiative and makes up a menu featuring “bacon and eggs”. It’s relatively easy for the chicken to provide the eggs… but serving bacon with this breakfast will literally kill the pig.

In life, some of us inevitably start out by being the idea-generators and playing the role of the chicken. We get excited about an idea and start working to achieve it, only to discover that the act of bringing this idea into reality may require that we become the pig, sacrificing an unrealistic amount of our personal time to make the idea a reality.

The problem is we need to remember that the “breakfast” requires ~~effort~~ work to produce. If we aren’t careful we place the burden of our success on the backs of our team and/or our future selves, only to get discouraged at the lack of progress along the way.

I do recognize there can be a happy middle ground where our ideas and our efforts to realize those ideas coexist. It’s just important to remember that realizing ideas requires far more effort than just coming up with the idea.

Iterative Personal Development

The road to greatness often comes with a lot of polishing.

This is how I write software. I start with a simple chunk of code and gradually write software components, refine code, and tweak the solution until it results in something that solves the requirements as gracefully as possible.

It’s awesome to realize this is also how we become better humans as well.

Personal example: I didn’t have to be the “Number 1 Dad” on day one. Being “Average Dad” was and is ok, so long as I keep working to polish my rough edges and stay focused in my drive to be great.

Be Transparent

My sister and I have chosen incredibly different career paths. She does graphic design for a multinational corporation and I… Well, I have chosen the path of writing software.

Over the years, it’s been fun to discover places where ideas from our professional lives intersect into each others’ fields. Two of these realizations came to me back when she was still in college / art school and have been among my guiding principles ever since.

First, one should never, ever, ever use Comic Sans. (Enough said.)

Second, and almost as obvious, the best way to get better at your craft is to encourage and accept constructive criticism.

For four very long years, my sister would repeatedly spend hours working on projects for her classes. She poured her heart and soul into nearly every project she was assigned, only to have her work ripped apart by her professors and peers in class the next day. A couple of times the criticism got to her but overall, she learned to accept and learn from the critical feedback she receives.

Criticism like this seems to make a lot of sense in the subjective world of creatives but… What about software design and development? — To answer with another question, riddle me this: If two independent teams are given identical non-trivial software requirements, how alike with their resulting implementations be?

Answer: Not much. 99.9% of the time the solutions will be unique. It turns out, software is a creative expression of the people involved in designing and writing it.

Why aren’t critiques a cornerstone of computer science education as well? — If we aren’t careful, it is often way too easy for software engineers to become attached to our code, holding it up on pedestals it doesn’t deserve.

In corporate settings, we often struggle to iron out inconsistencies in code bases. We create Coding Standards to make code look consistent and try to conduct Code Reviews to discover problems and enforce the coding standards. Unfortunately, my personal experience has shown that the code reviews and coding standards are difficult to implement effectively. Some of the many reasons include:

there’s never enough time to conduct thorough reviews
code reviews are boring and it’s always more fun to chase new things
complicated code can be hard to follow
leaky abstractions are… leaky

Reviewing software is often harder to do effectively than the act of writing it in the first place. Wait. This is important. Let me write it again, this time in italics. Reviewing software is often harder to do effectively than the act of writing it in the first place.

I believe code reviews hold incredible promise. There is no better way for us to learn than from each other.

I’ve struggled for years to come up with a solution to the code review problem. If you are on a team of software engineers, perhaps this isn’t a problem for you. Perhaps you are able to conduct thorough reviews on every single line of code in meetings and/or with sophisticated tools.

I have not been so lucky. For a large part of my career, I’ve worked as a solo software engineer on teams of electrical engineers. — If you think it’s difficult to conduct code reviews with software engineers, just try to get an electrical engineer to pour through thousands of lines of source code.

The best sustainable solution that I’ve been able to come up with is to architect the software to facilitate “targeted reviews” while also providing the complete source code to my teammates in an easy-to-access intranet-based folder. Relevant sidenote: Distributed version control repositories (e.g. Git, Mercurial) are great for publishing source code in intranet folders because the tools make it easy to automate the updating of external locations with a single shell script.

Doing all of this supports the targeted reviews but it often requires PowerPoint “conversation starters” to introduce key algorithms before ever looking the actual code. When done right, coordinating brief meetings with the relevant parties can be straightforward.

Overall success of this approach hinges on the act of being transparent with every bit of produced source code and supporting material; all organized in a way that is discoverable to the reviewers. The good news is this form of transparency has an added benefit of building credibility.

When you are on a team, the source code and opportunities to criticize the codebase should always be available to everyone on the team anyways. The benefit is in actively soliciting feedback.

Consider The Customers

It often surprises me when my fellow software engineers seem to forget our customers when designing software.

There are at least three major customers for any piece of software and it pays to remember their needs with every software project.

The first group of customers are our bread-and-butter. They are the users of the software; the humans that will need to discover and rediscover how to make it work for them. It can be a lot of work to keep users happy but the developers’ reward is certainly there if you can create a product they can use effectively. Satisfying users gives you an opportunity to literally make their lives easier. When we focus on the users, we are literally adding value to their lives.

Our second, equally important customer is the person or entity that is actually paying for the software. If you are lucky they are the same as our users but often, we are not so lucky. Examples can include: the enterprise software decision makers, our bosses in the mega-corps, and/or those incredibly amazing users that are actually willing to pay for the tools they use.

Together, the first and second groups of customers are pretty darn important and it pays to keep them happy. Without those customers, there wouldn’t be any Twinkies in our desk-drawers or beer in our ~~mini fridges~~ home refrigerators. They keep us employed.

And yet! I am asserting that there’s a third customer and this group of customers is often neglected the most. It’s a case of the “Cobbler’s Children Have No Shoes“. Our third customer is the software engineer that will need to come back and evolve the software product and/or fix our mistakes.

This engineer is often a future version of ourselves that have completely forgotten the nuances of the code after having been away from it for six months or a year. Other times, it is a teammate that has been tasked with maintaining an existing design. I am a firm believer that “code is read much more often than it is written” and believe it is important to remember that someday, someone will be looking at every line of code I write.

I don’t know about you, but if I can help it, I would like to limit the amount of bad code I have to defend when I arrive at those Perly Gates.

Sure, it can be fun to develop clever solutions to a problem but if the maintainer of the software isn’t able to discover how to use or modify the software in a reasonable amount of time, then the solution is a failure.

There are a number of ways to help “the next guy” including detailed commenting and well-structured, well-thought-out software designs. The important thing is to actually remember these ~~customers~~ maintainers every time we write even a single line of code.

Problems inevitably arise when we put ourselves above our customers. When we develop products using short cuts or don’t consider all of our customers’ needs, we look unprofessional and our customers suffer.

Who Needs Goals?

A while back, while working for a former employer, I was called into my manager’s office. It was “annual review time” and my manager had decided one-on-ones were the order of the day.

The review itself went well. I had great input from my teammates and had a green light for a promotion in the next six months (awesome, right?). Unfortunately, there was a problem.

My manager looked me in the eye and asked: “What are your career goals?”

I honestly did not know. The truth is, I had no career goals. And to both of our horror, I even expressed this thought out loud.

I was working there because it seemed the logical thing to do. I was learning a lot, had interesting challenges to work on, and was part of a team that got things done. Now my manager was telling me that I also needed to manage my “career”?

This was a pivotal moment for me. It helped me realize that I was on the wrong path. Quite frankly, I wasn’t on any path at all. I had my head down and was only focused on the day-to-day.

In life, we are often presented with a baseline of blessings. Things like: generally good-health, hopefully a steady paycheck, and in my case, stunningly good looks. It’s a head start.

The next step is to dream about what you want your life to look like in the next X years (5 years? 10 years? 50 years?). To be honest, I typically have a tough time with this step but! it’s incredibly important.

There are a bunch of directions you can go with goals. They don’t always have to be career focused but focusing on a career can be a great start. What are you looking to achieve? Technical competence? Financial security? Weight loss? Launch a business?

While it is good to dream big; it’s also important to remember that goals will remain unrealized until action is taken to achieve them. Wandering through life, waiting for life to happen to you lacks purpose and typically results in unfulfilled goals. This would be a waste of your ~~incredibly good looks~~ incredible blessings.