been trying to learn more about neural nets in the past two days. a lot
of the math flew over my head (linear algebra? multivariable calculus?
classes I have yet to take).
but i'll just jot down some notes and things that i've taken away, i.e.
what little my non-porous nut of a brain has absorbed (i'm going to say
'i think' a lot here, i think):
neural networks are modelled after real processes that happen in our
brain (cog sci). a nn is comprised of at least 2 layers, the input layer
and output layer, and in between there could be any number of "hidden
layers". the layers themselves comprise of "neurons".
it is often said that the more hidden layers there are, the better. same
goes for neurons in those layers. so what is optimal? i'm guessing it
has to do with time/space complexities as usual. we could have a billion
hidden layers to really fine-tune the nn, but at the cost of tremendous
time/space overhead. but this is just my hypothesis. and perhaps that's
why optimising the right number of layers/neurons is also a task in
itself.
for the next few steps, there is math involved. lots.
the inputs are multiplied by specific weights, and summed together.
then, a bias is added, and the entire thing is passed through an
activation function, of which there are multiple (sigmoid, tanh, relu).
these weights are used to determine the "importance" or "impact" of the
individual neurons. the bias is apparently used to shift the resulting
sum into a "normalised" range of sorts. the activation function is to
give the nn more complexity, without it, it would simply be a linear
regression of sorts. these parts i'm most shaky about. i'm jumping ahead
of myself, but are these just terms/mathematical properties sprinkled
into the nn that give it the ability to be flexible/trained/modified?
i'm curious to find out how these were materialised in the first place.
anyway, this sum is then passed into *one* neuron of the next layer
(hidden or output). to backtrack, i'm not sure if the weights differ
from one neuron to the next. there's a lot to keep track here.
the input data is passed through all the layers, and this process is
known as forward propagation. it's used when training the nn and of
course when we actually pass in unseen data that we want the nn to
process (usually classification, see XOR or mnist, the 'hello worlds' of
NNs).
how does the machine "learn"? how does the NN get better?
backpropagation. this is where the nn fine tunes its weights such that
subsequent outputs are closer to the target/actual outputs (this is why
training data is important, with correctly labelled test data).
the difference between training output and target output is known as the
cost error, i think. usually derived using the MSE, something like
1/n*Σ(output-target)^2 (where n is the no. of output nodes?). then to
fine-tune the weights, here comes multivariable calculus and gradient
descents, realms which i have no hope of surviving in atm, but still
useful to note down for future reference.
the trouble is, i have such a difficulty not only learning how they
actually train the nn, but also the math itself, so here are some notes
on that in the form of chicken scratch and babble:
rinse and repeat and you have a nice neat NN. I think.
i tried learning to program one in C++ from scratch. i can understand
the programming, and the implementation, but my understanding of what
and how the nn does it is too shallow for me to shamelessly own that
piece of knowledge. i'll try again... someday...
final notes: