Reading Tim Harfords Adapt got me thinking about the agile practice of continuous improvement which comes from lean manufacturing. Harford talks about systems having in built faults and humans add fail safes to eliminate pilot error. This creates highly complex systems that are nolonger easily understood and the fail safes themselves create unintended consequences.
A minor addition of a failsafe could cause catastrophic failure if a perfect storm of states occurs. And as black swan theory suggests it will.
Tweaking complex process can be fatal, do so with care.