Why is it necessary?
Machine learning systems are statistical in nature and, on occasion, make mistakes. Additionally, they are prone to concept drift, i.e. changes in the incoming data. However, how to implement fault tolerance or error detection as its sub-task is not always straightforward or obvious [2]. Error detection is especially difficult when it is difficult to pinpoint an input being out-of-distribution, or the output does not directly violate any rules set for the outputs. Thus, there is a need for error detection measures that monitor the inner workings of the models, when possible, to detect unwanted or suspicious behaviour.
How does it work?
For the nodes in a neural network, a co-activation rate is calculated using the training set used to train the neural network. Co-activation rate is the rate with which a node activates together with another node in the training set. In other words, it describes which nodes tend to activate together with inputs the network should be able handle. Inversely, low co-activation rate tells which nodes usually do not activate together. The number of these rare co-activations is higher in out-of-distribution inputs and incorrectly predicted ones than in correctly predicted inputs [1]. Thus, monitoring the number of rare co-activations and the changes in the numbers shows potential in error detection and monitoring potential concept drift.
