I was bored in my stats class today and so decided to ''simplify'' some basic neural nets and I found that when the activation function of all of the neurons in linear, I tested f(x)=x but assume that its true with all linear f(x)''s, the net will always produce linear results. I found this by finding a general equation that gave the output of a basic NN given the inputs and weights. Because the weights are constant, you can simplify this massive equation into some much simpeler linear form. Is what I found what other people have found? Basically I just want to know if what ive proven is acurate and if all of y logic is valid. Thanx tazzel3d P.S. this means that the only way to model nonlinear data is through nonlinear activation functions... What functions have you used? Thanx again

Yup. That is completely true. You lose the advantages of multilayer networks when linear activations are used. Feeding a signal though more than one layer of linear activations is no different then what could be obtained from a single layer.

- mongrelprogrammer

Hey, thanx for the confirmation. Though i''m still confused about what other activation functions to use... What do you guys use?

thanx for the help!

If you train your network using a gradient descent technique then you will usually get better performance from using tanh activations in the hidden units and logistic ones in the output units. I believe the comp.ai.neural-net faq has further details.





ai-junkie.com