2

I'm noticing something strange, i followed an example of coding a fully connected neural network in 3 layers. it uses backpropagation, and it works great. For example using Sweep optimization (ea sweeping trough starting train variables) this program can resolve the Irish flower data set with 99.16% accuracy (as total of train and validation data. I think thats an extreme score and its even so that i can bread multiple neural nets with such a score. So those networks are working great. (standard neural nets like 3:4:3)

But i wanted to put them trough other tests as wel. Then i thought lets try calculating RGB to HSL color space. Turns out the networks score 100% if try
RGB to H
or RGB to S
or RGB to l

But one network doing it all ea RGB to HSL ... it seams impossible. Is there something fundamentally difficult different that makes that problem that different. I would have thought that the nodes would weight balance and eventually seperate channels binding right output to right hidden nodes. But somehow it doesnt happen even if i triple the hidden nodes.

the code i use is based on this code in case your interested https://gist.github.com/atifaziz/9462430 (although it hasnt the sweep optimizer probaply reaching 85% accuracy).

I'm new to neural networks, and i like to know if there is some specific reason as of why a 3 layer neural net seams hopeless in resolving RGB to HSL Where it takes random RGB values to output HSL values; while having only one output for H or S or L result in 100% resolving. Is RGB to HSL somehow a different category problem. It doesnt seam to matter how many hidden nodes i use (but i only have 1 hidden layer, i'm not yet understanding the coding math behind deep neural networks).

So in short is there some mathematical reason relating to the backpropagition method. that a 3 layer network (input-hidden-output) can not be resolved ? As I am curious to know why the network cannt resolve it.

Sweep training that i use takes hundreds of starting variations, and each of them gets trained 5000 epochs which is more then enough for the irish flower set (thats resolved within 100 epochs and optimal in 1000 epochs).

(better explained i want to know if there is some fundamental reason why it doesnt get resolved)

Peter
  • 131
  • its still on hold ? i updated my question i think someone in this field would understand what i say. (but if your not in neural networks math then probably not) – Peter May 31 '17 at 09:07
  • This question might be more appropriate on a different stackexchange site which is specific to machine learning or neural nets. I don't see a specific example of the math you're not understanding. If there's an equation or problem (a math problem) then please update your question accordingly. – Neev Parikh May 31 '17 at 10:00
  • i'm not aware of more specific sites, as i think it comes down to backpropagition solving. where each out put node changes all hidden layers in a fully connected network, And so i wonder if its logically speaking impossible to reach a solution in which each output node reserves only a few hidden nodes. Is it a theoretical limit of BPP. since is the BPP neural network is based on error correction statistics i wonder if some mathematician might know this answer. – Peter May 31 '17 at 11:55
  • I know the answer, but you may not like it. The answer is that currently we do not have any real AI, and hence all so-called AI can never be more intelligent than their creators. They may only appear so to laymen because they can perform calculations much faster than humans, but the actual intelligence is lower. In particular, although mathematical theorems guarantee eventual convergence of a sufficiently large neural network to the desired function, the bound on the time needed to converge is astronomical for any reasonably complicated function even if humans can easily figure it out. – user21820 Jul 24 '20 at 05:25
  • More specifically to the problem you state, you are trying to get the neural network to learn (without human constraining) that the outputs can be individually computed, and you want it to eventually internally have 3 separate parts, one for each output. But although that is what a human can see, it is not what a typical back-propagation mechanism can 'recognize'. Similarly, you know (by your own human reasoning) that there is a solution with 3 times as many hidden nodes, but the neural network does not know that, especially if you have more hidden layers. – user21820 Jul 24 '20 at 05:32
  • Remember that for each desired function there is a minimum number of layers and nodes per layer that a neural network must have to be even able to simulate that function. At the same time, having more nodes per layer greatly increases the time needed to converge to the solution, and having more layers yields an even more drastic slowdown of the convergence. In the end, you'd have to use your human intelligence to choose the best parameters, or at least design a heuristic or meta-heuristic to find the best parameters. – user21820 Jul 24 '20 at 05:37

0 Answers0