System evaluation using fault and error injection

Computer Engineering Research Center
The University of Texas at Austin

A major problem in the development of fault-tolerant systems is the accurate determination of the dependability properties of the system. Unlike performance, which can be evaluated through the use of benchmark programs, the degree of fault tolerance and reliability of a system cannot be evaluated in such a manner, since we do not often have the luxury of allowing systems to run for many years to see their behavior under fault effects. The generally accepted solution to this problem is to inject the effects of faults in a simulation model or a prototype implementation, and to observe the behavior of the system under the injected faults. Fault injection in a simulation is very flexible but far too time consuming. On the other hand, it is much more difficult to inject accurate (i.e., realistic) faults into a prototype, but the effects of faults on operational code can be readily observed.

We have developed powerful techniques for emulating hardware faults using software. These were incorporated into a flexible fault and error injection system, FERRARI. We modify the executable image of a program so that when this modified code is executed, the behavior of the system would be the same as if the internal fault had been present. This approach would provide the desired flexibility, and at the same time, would allow us to execute many experimental runs in a relatively short time. FERRARI is based on the public-domain GNU tools, and the first implementation was on SPARC stations.

Many commercial applications, however, use proprietary compilers and debuggers, for which the source code is not available. We recently developed a new fault injection system, FIESTA, geared to embedded COTS systems. This injection system is implemented under the real-time operating system, VXWorks, with the target system currently an embedded Motorola 68040 controller. Faults can be injected into applications compiled with GNU tools as well as the GreenHills ADA system.

G. A. Kanawati, N. A. Kanawati and J. A. Abraham, "EMAX: A High Level Error Model Automatic EXtractor," Proceedings AIAA-93, San Diego, CA, October 19-21, 1993, pp. 1297-1305.


This paper presents EMAX, a High-Level Error Model Automatic EXtractor. EMAX simulates all user-selected, low-level faults (at the gate and/or the switch level) that may occur inside a processor chip, and generates the error output patterns produced by the faulty circuits. These generated patterns are used to extract high-level error models. When these error models are further analyzed, a sequence of instructions can be derived which, when executed, produce the same error patterns as those obtained when simulating the hardware with low level faults. When this sequence of instructions is fed to a software fault and error injection tool, it willl allow the use of accurate and cost-effective higher level fault/error injection patterns for validating the dependability properties of a system. Error models extracted for an example processor are presented and are analyzed.

G. A. Kanawati, N. A. Kanawati and J. A. Abraham, "FERRARI: A Flexible Software-Based Fault and Error Injection System," IEEE Transactions on Computers, vol. 44, no. 2, February 1995, pp. 248-260.


A major step toward the development of fault-tolerant computer systems is the validation of the dependability properties of these systems. Fault/error injection has been recognized as a powerful approach to validate the fault tolerance mechanisms of a system and to obtain statistics on parameters such as coverages and latencies. This paper describes the methodology and guidelines for the design of flexible software based fault and error injection and presents a tool, FERRARI, that incorporates the techniques. The techniques used to emulate transient errors and permanent faults in software are described in detail. Experimental results are presented for several error detection techniques, and they demonstrate the effectiveness of the software-based error injection tool in evaluating the dependability properties of complex systems.

N. A. Kanawati, G. A. Kanawati and J. Abraham, "Dependability Evaluation using Hybrid Fault/Error Injection." Proceedings IEEE International Test Conference, Erlangen, Germany, April 24-26, 1995, pp. 224-233.


This paper presents a new hybrid fault/error injection technique which overcomes the limitations of both software-based and hardware-based approaches. The logic for the hardware fault injection circuitry is implemented using Field Programmable Gate Arrays, and the software is an extension of FERRARI, the software-based fault injection system. The combination of these techniques allows the incorporation of new capabilities by the use of mechanisms to trigger and synchronize the injection of a fault or error with events in the system. Results of physical fault/error injection experiments on a SPARC1 system are presented. The injection was synchronized to the executing modes and load conditions of the system. These results show that the system behavior is very sensitive to the internal state and load. Therefore, in order to validate the dependability properties of a system, it is imperative to inject faults/errors while the system is in critical conditions and different execution modes.

Back to: