We review problems with IT and the legal factors that make it hard for even poor IT systems to be called into question, a problem that leads to harms and miscarriages of justice — and failure to learn the right lessons and improve. After reviewing the history of rigorous approaches to software development and current best practice, we make realistic proposals that (if adopted) could mitigate and prevent these problems. Our proposals should also be used to stimulate research and debate how to address the serious consequences of society’s unjustifiably casual approach to the most complex and pervasive invention it has ever relied on.