Deep Neural Networks Classification via Binary Error-Detecting Output Codes

Abstract

One-hot encoding is the prevalent method used in neural networks to represent multi-class categorical data. Its success stems from its ease of use and interpretability as a probability distribution when accompanied by a softmax activation function. However, one-hot encoding leads to very high dimensional vector representations when the categorical data’s cardinality is high. The Hamming distance in one-hot encoding is equal to two from the coding theory perspective, which does not allow detection or error-correcting capabilities. Binary coding provides more possibilities for encoding categorical data into the output codes, which mitigates the limitations of the one-hot encoding mentioned above. We propose a novel method based on Zadeh fuzzy logic to train binary output codes holistically. We study linear block codes for their possibility of separating class information from the checksum part of the codeword, showing their ability not only to detect recognition errors by calculating non-zero syndrome, but also to evaluate the truth-value of the decision. Experimental results show that the proposed approach achieves similar results as one-hot encoding with a softmax function in terms of accuracy, reliability, and out-of-distribution performance. It suggests a good foundation for future applications, mainly classification tasks with a high number of classes.

Explainability as a Method for Learning From Computers

“…The following theorem discusses this property. It is an extension of Theorem 1 in [17] to the ternary codespace. Theorem 1.…”

“…• Use them directly as used by the non-linear classifier 𝑦 𝑖 = 𝑓 𝑖 • Use features relevance measure 𝑦 𝑖 = 𝜌(𝑓 𝑖 ) = 𝑅 𝑖 . In both cases, the feature or its relevance needs normalising to a unit interval to interpret its value as the truth value of the fuzzy logic statement [17]. For the values, we normalise 𝑦 𝑖 ∈ ℛ, 𝑖 ∈ {1, … , 𝑀}, and use min-max linear normalisation:…”

“…) there is only one codeword with a truth value higher than the upper limit, (c) rounding relevant bits according to (4) gives the winning codeword. The proof of the theorem is omitted due to its similarity to [17].…”

“…The basis of decision explanation from features can be Aristotelian (or fuzzy) logic or linear systems. Paper [17] studied the implementation of a classifier directly by a fuzzy logic function. We add a classifier formed by a fuzzy logic function using Zadeh's fuzzy logic parallel to the neural network classifier and restrict the description of the solutions to explainability in forwarding DNNs for pattern recognition since the proposed solution is also of this type.…”

Comparative Analysis of Load Forecasting Models for Varying Time Horizons and Load Aggregation Levels

“…For the temporal and calendric feature categories, we use two different approaches. In the first one, we generate one-hot encoded values [24] of the current hour (h i , i ∈ {0, 1, ..., 23}, weekday (d i , d ∈ {1, 2, ..., 7}) and month (m i , d ∈ {1, 2, ..., 12}). In addition to that, we generate two cyclic features by applying the trigonometric functions to the hour of the day and the month of the year.…”