Compact Neural Network: Parameter Reduction using Sign Combinations

One problem of using neural network to learn real life data is that there are too many parameters (weights) to tune. Here, a new method of reducing the number of parameters is proposed. The basic idea is to represent individual nodes by vectors of different combinations of positive or negative signs. Nonlinear relationship between the nodes can therefore be encoded in their sign combinations. This Compact Neural Network can implement Parity without hidden nodes. It can also achieve the same generalization performance on real life data using a fewer number of hidden layers and nodes. More importantly, this network does not need heuristics (e.g. manually design a polynomial form as in functional link neural network) and can be scaled up easily to become a deep network for solving very complicated classification problems.