Why I do NOT believe in Error Backpropagation

16/04/2015 11:58

Blog type : Research


Why I do NOT believe in Error Backpropagation

Neural Network for Real World Data Classification

and Namely in Intelligent Robotics ?


Abstract : This small note gives an opinion about neural network tools as classification mapping machines. The Mapping is from the feature space into a Class space. My opinion is that classification using Error Back-propagation neural networks are not able to calculate very complex discrimination hyper planes in feature space especially in case if a class consists of more than one cluster in feature space. Therefore I do advocate for ART-like neural networks which have different approach to feature space segmentation into class space. This my views are provided using Question and answers to above topic.


Peter Sinčák

Background: Patter Recognition, Image processing, neural networks, Intelligent Robotics, Cloud Robotics, Cloud based Pattern Recognition


Question #1 : What we are doing if we are training feedforward neural network with Error Backpropagation adaptation Rule on classification task and training set consisting of Input and Outputs ?

Answer : In fact neural network calculates a discrimination hyper plane which can be based on data sometimes very complex hyperplane and calculation of hyper plane equation can be time consuming procedure. On the following figure is a simple 2 dimensional feature space and 2 classes organized in 2 cluster each class – blue and read class. Green is a possible discrimination hyper plane. 





Question #2 : What we are doing if we are ART like neural network on classification task and training set consisting of Input and Outputs ?

Answer : ART like neural networks are discovering clusters in the feature space and the cluster identification is computationally totally different procedure as calculation of equation related to discrimination hyper planes. The difference of this procedure is visible on the following figure. Please compare it with the previous figure representing feedforward neural network with Error backpropagation adaptation approach. So make opinion which equation is less complicated and easy to compute.



Question #3 : What if we have created a functional classification machine based on feedforward neural network with Error backpropagation adaptation rule and we additionally decide to add one more class for classification. How are can solve this problem using feedforward Error backpropagation ?

Answer : In this case we need to recalculate a new situation from scratch. This is very important point and in fact this approach confirm that these neural networks are not able for incremental learning. This recalculation can be sometime, depending on data, very complex and rather complicated. In fact we have to calculate a union of discrimination function including a new one for a new yellow class.



Question #4 : What if we have created a functional classification machine based on ART like neural network we additionally decide to add one more class for classification. How are can solve this problem using ART Like neural network?

Answer : In this case I think situation is rather more simpler comparing to the above approach by feedforward neural network. In this case we retain the previously computed clusters and we just add one more cluster of a new class (if a class really consists of one class) and calculation of this new cluster is not so complex as in previous situation. The approach in fact confirms incremental learning ability of ART neural networks and also representing more sophisticated approach to feature space segmentation. Segmentation procedure is representing classification machine with incremental ability.


Question #5 : So what is more suitable tool for creating a classification machine in hyper dimensional feature space ?

Answer : Since we are not able to analyze if the classes  are mono-clustered or multicluster in feature space - I strongly believe that for classification tasks a ARTlike neural networks are more suitable. Also incremental ability of learning in ARTlike neural networks over re-training a new class in Feedforward neural network is beneficial for ARTlike over feedforward neural networks.


Question #6 : Today is very popular Deep learning neural network. Sometime this approach is connected to feedforward neural network with Error Bacpropagation. Is it true ?

Answer : No it is not true. The predecessor of Deep Learning are Convolutional neural network. In fact these are hierarchical neural network which consists at least of 2 parts – feature space transformation and feature extraction (creation of new feature space) and classification. The main issues of classification is an input labeling. The incrementally  option of Deeplearning is not mentioned and in case if we want incremental Deep learning we do need to replace a second part of Deep learning by ARTlike neural network which is not very common. More common is e.g. to replace a second part of Deep learning by Support Vector machine for classification procedures hence it still does not solve a incremental aspect of classification.


Question #7 :  Where is an incremental learning is important if so ?

Answer: A good example is Robotics. It seems that in social learning which Human interact with Robot a so called “deictic learning” related to deictic gestures is absolutely essential. In this approach a human is gradually pointing to different objects and Robot is learning gradually new and new objects, situations, procedures and etc. If we assume that some group of robots interacting with different people can share and build a common knowledge base – we are creating fast and crowdsource incremental and shared knowledge for robots and this is a big and complicated challenge.


Question #8 : What is the most important challenge in pattern recognition and its application in Intelligent Robotics ?

Answer: there are many interesting challenges in pattern recognition and mainly in connection with intelligent robotics. I do think that the main issue and challenge to solve the question - how to work in dynamical feature space. Working in static feature space is rather big obstacle and humans are working in dynamical feature space. If we would be able to solve this problem the classification procedure will be more straightforward and simpler.


Question #9 : What is a future of Cloud Robotics and off board computing for Intelligent Robotics ?

Answer: I think the Cloud computing has super features to be used in Intelligent Robotics. We are living on the edge of 5G Wireless networks and now in 4G we are achieve aprx. 35 Mbit data transfer which is rather sufficient for many data transfers. So basically in Intelligent Robotics we are heading towards BIG DATA and software agent will be doing its job for various social robots. Certainly some computation will be on board usually those procedure which are required to operate in super real time and movement. This Cloud Robotics is also important for crowdsourcing and  knowledge sharing point of view however we do not have any clue how will be a knowledge for Robots stored and how handle obtained knowledge for different type of robots with different capabilities.


Question #10: What is importance of  intelligent Space and will we have a Robot Ready environment ?

Answer : I definitely think that Intelligent Space is rather important for robots. There is a notion called “networked Robots” and these robots are using external sensors, cameras to be able to make a right decision and NOT only sensors on the Robot but also outside robots and even information on the net. So I believe we are expecting a large revolution in robotics towards internet of things, BIG Data and Cloud Collective Intelligence. All these aspect are totally unknown and humans must face these challenges for their benefit and benefit of all mankind. 


Peter Sincak