Ignoring things won’t make them go away. It only makes it harder to face them when they finally come around. — A. Meredith Walters
Failures during Product Development
During product development, teams encounter several kinds of failures. A component may fail during testing. One or more prototypes may miss performance specifications. A software unit may not function as expected, or the system may crash under certain conditions.
As we explored in Commandment 6 (“Conquer Fear”), the psychological fear of failure can paralyze a team. In Commandment 8 (“Pay Obsessive Attention to Detail”) we emphasized “Test to fail”. This commandment is about how to manage component failures when they are found.
Customers expect a product that works reliably in use. This applies not only to catastrophic failures, where the product stops functioning or creates a hazardous situation, but also to products that fail to meet specified performance levels. Poor performance has a significant impact on customer satisfaction and is unacceptable. Software products entering a nonresponsive state is a strict no-no. Any subsystem or component failure noticed during product development must be treated as a warning of a similar failure during the operational life of the product. It is much easier and cheaper to act on product failures during development than to pay for failures in the field.
Every unresolved development failure creates technical debt. The debt may eventually be paid through warranty claims, customer dissatisfaction, support costs, lost sales, regulatory exposure, or damage to reputation. This has much higher long-term consequences.
Prevent failures
Designing to prevent failures is one of the most important aspects of product development. There are several techniques available, such as “Worst Case Design”, “Robust Designs-Taguchi Method,” and “Design Failure Mode and Effect Analysis (DFMEA)”. In software, preventing failures means practicing Chaos Engineering: injecting high network latency, dropping packets, feeding malformed JSON payloads, and running database or memory stress tests to see how gracefully your application degrades under pressure. These techniques minimize failures caused by known failure mechanisms. Despite all precautions, it is highly likely, almost inevitable, that some failures will be discovered during the product development process at various phases. It is especially important to investigate all failures during a product development project.
Investigate and analyze all failures
Sometimes, there is a tendency to dismiss a failure as a “random failure”. It is possible that a component is marginal and outside its specification limits under some conditions; however, with a well-chosen component, this should have an exceptionally low probability. The sample size during development is too small to catch such a component, which already has an extremely low probability of occurring. One of the most dangerous phrases in software development is, “It works on my computer”. It is important to ensure that the software works in all specified computer environments, not only on one setup. A failure ignored as random during product development will reappear during manufacturing or in field use. Such situations, when known, must be fixed before release of the product.
Evaluate whether the component is being “stressed” during some situations or use cases beyond the limits specified for the component. This can happen more frequently, in the case of e.g., electronics power devices, mechanical load bearing components etc., in some combinations of inputs or loads. Components may suffer degradation because of faulty assembly techniques, or handling. Careful study of the component data sheets, all possible operating conditions, and computing infrastructure is necessary.
Evaluate tolerance and safety margins carefully. One may not always find components at the extremes of tolerances for testing. It may be necessary to simulate worst case tolerances. The exact method of evaluation of tolerance limits and operating conditions will depend on the specific conditions. Software must be tested under all the specified computer configurations.
For budding developers especially, it is easy to fall into the psychological trap of dismissing a glitch as a ‘one-off’ quirk of the local environment. Remember: hardware or code does not behave randomly. If a failure happens once during development, assume it can happen again in production until proven otherwise.
Imbibe a simple failure investigation discipline:
- Reproduce the failure.
- Record all conditions.
- Form hypotheses about the cause.
- Design tests to prove or reject hypothesis, one at a time.
- Verify root cause, check that failure does not happen after removing the cause.
- Confirm that failure reappears after re-applying the cause.
- Document and track corrective action.
- Retest.
Act on failures
All the failures found at whatever stage must be recorded and tracked till closure. A “Failure Reporting and Tracking” system must be an integral part of the product development process. A failure is not closed when it disappears; it is closed only when its root cause is understood and corrective action has been verified.
It is necessary to take cognizance of all the failures seen during development, and act to resolve them. Not every failure must be corrected immediately, but every failure must be investigated, understood, documented, and consciously dispositioned. Ignoring is not the same as prioritizing.
Teams do not ignore failures because they are careless; they ignore failures because they are under pressure from deadlines, budgets, customer commitments, and schedule slips. Leadership has an important role to play here: it must ensure that no one in the team is tempted to hide a failure, and that everyone feels free to report a failure as soon as it is noticed. A structured risk management system plays an important role and may be mandated by regulatory requirements. When a failure is reported, it must be consciously dispositioned. One useful adaptation of the MoSCoW prioritization method is to classify the required action as Must, Should, Could, or Won’t Fix. Even a “Won’t Fix” decision is acceptable only when the failure is understood, documented, reviewed for risk, and consciously accepted. The list below defines each disposition and provides real-world examples across various engineering disciplines. How a failure is categorized as Must, Should, Could, or Won’t Fix depends heavily on the domain, operating context, and intended use. For instance, an issue considered a ‘Should Fix’ in a consumer hand tool may become a non-negotiable ‘Must Fix’ in a surgical instrument, where even a minor distraction to the operating surgeon can compromise patient safety.
- Must fix: The root cause must be found and the failure corrected before release. —An internal power rail shows a periodic voltage drop during transmit bursts, causing a microcontroller reset under low battery conditions. It directly degrades core functionality and leads to unacceptable field reliability issues.
- Should fix: The root cause should be found and corrective action planned at an appropriate stage. –A plastic housing latch emits an audible creak when pressed at an off-angle, though the latch mechanism holds securely. It affects the perceived tactile quality of the product, but does not compromise mechanical integrity.
- Could fix: The root cause may be investigated and corrected if time, risk, and priority permit. —The ambient cabin lighting in a car takes 0.5 seconds longer to dim when entering a tunnel if the automatic headlights were toggled manually right before entry.
- Won’t fix: The failure is consciously accepted after risk assessment, documentation, and approval. —An ECG monitor shows temporary signal noise when operated within 0.5 meters of an active high-frequency electrosurgical cautery unit. The behavior is documented in the risk file, verified not to damage the device, and clearly addressed in the user manual through separation-distance instructions.
The critical point is that every known failure must pass through a deliberate decision process; none should disappear through silence, memory, or schedule pressure.
Systematic “Root Cause Analysis” of failures is necessary in order to act on them. Techniques such as “5 Why” and “Fish Bone Diagram” may sometimes be useful, but they are more effective during the manufacturing stage. I have found that “Boundary Condition Analysis” or “Failure Mode Journey Mapping” is useful. Start by imagining what could have happened and then follow the logical chain of events to identify possible consequences. For the component or subsystem that has failed, it is very useful to form a hypothesis for the root cause of the failure.
Invite Failures
It is not good enough to wait for the failures to surface. Failures are your best friends. Invite them if they do not pay a visit! Create worst case situations, like low power supply voltage, high or extremely low temperatures, high humidity, conditions even more severe than allowed by the product specifications. It may be all right for the products to not meet full specifications in extreme operating condition, but the products should not fail and should recover decently. Many times, products fail if some components operate at the extremes of their specifications. Such components operating at or near extremes are rare.
Learn from failures
Investigation and thorough analysis of observed failures is not only beneficial for the product under development. It also benefits future product developments, contributing to improvements in underlying development processes, such as component selection, defensive software and hardware design, safety factors and tolerances considered, to name a few. Such findings should be appropriately documented and disseminated within and across product development groups. Documents such as guidelines, formats, SOPs should be updated as required.
When I was an electronics designer, if I found a failure because of a component operating at the extreme of its tolerance limit, I used to preserve that component properly labelled. It could be used in testing future products. Other colleagues used to borrow such components to test their designs and contribute components found by them to this collection of troublemakers. This collection became invaluable to the lab.
From a Minor Glitch to a Major Business Failure
Hotels in the past relied on “piped-in music”, a central system in which multiple audio sources played channels that were distributed round-the-clock through thick bundles of cables to rotary switches in every room. The system required massive cabling and constant maintenance.
Our Industrial Communication group engineered a brilliant semi-digital solution using a single cable line and compact room receivers, significantly cutting costs. We built a prototype, and it worked wonderfully. However, during testing, a single room unit failed. Unable to quickly isolate the root cause, the team shrugged it off as a “random component glitch” and pushed forward.
We offered the system to hotels on a trial basis. Within six months, over 20 percent of the room units failed. Trial hotels lost confidence and retreated to legacy cabled systems. Leaving guests without background music was unacceptable. Our sales team lost faith, and we were forced to pull the product entirely.
While we were sad about this debacle, we continued looking for the root cause. We were left with one vital clue: all failures occurred when rooms were unoccupied. We finally traced the failures to a power-down state transition: the room’s master power switch was cut off, but the input coming from the central system was still active. We eventually engineered a fix and used it successfully in different products developed later, though an entire market segment was permanently lost.
A bitter lesson: Never ignore a development failure as a random anomaly. A small unexplained bug in the laboratory can easily become a catastrophic failure in the marketplace.
Summary and Key Takeaways
Any failures of subsystems or components noticed during product development are pointers to field failures and should never be ignored.
- Failures should be prevented using several techniques available to the developers.
- If any failures occur during development, as they inevitably will, they must be analyzed and corrective actions taken.
- Every failure presents a learning opportunity, which can be very useful for future product developments.
True engineering maturity begins the moment you stop treating failures as annoyances to be bypassed and start treating them as vital clues. Every bug, crash, or margin slip is your product trying to tell you where its boundaries lie. Listen closely, investigate relentlessly, and never let a failure go unexplained.
Have you ever ignored a “random” bug during development, only to watch it blow up in production? What is your team’s protocol for tracking elusive edge cases? Share your stories in the comments below.
In the next and final commandment, we will explore a deceptively simple principle: Go and See.
Failure is an option here. If things are not failing, you are not innovating enough. -Elon Musk