lec 18 NEURONS

===== [The field of artificial intelligence started with some wrong premises. The early AI researchers attacked problems like chess and theory proving, because they thought those exemplified the essence of intelligence. They didn't pay much attention at first to problems like vision and speech understanding. Any four-year-old can do those things, and so researchers underestimated their difficulty. Today, we know better. Computers can effortlessly beat four-year-olds at chess, but they still can't play with toys nearly as well. We've come to realize that rule-based symbol manipulation is not the sole defining mark of intelligence. Even rats do computations that we're hard pressed to match with our computers. We've also come to realize that these are different classes of problems that require very different styles of computation. Brains and computers have very different strengths and weaknesses, which reflect these different computing styles.] [Neural networks are partly inspired by the workings of actual brains. Let's take a look at a few things we know about biological neurons, and contrast them with both neural nets and traditional computation.]

• CPUs: largely sequential, nanosecond gates, fragile if gate fails superior for 234 x 718, logical rules, perfect key-based memory
• Brains: very parallel, millisecond neurons, fault-tolerant [Neurons are continually dying. You've probably lost a few since this lecture started. But you probably didn't notice. And that's interesting, because it points out that our memories are stored in our brains in a diffuse representation. There is no one neuron whose death will make you forget that 2 + 2 = 4. Artificial neural nets often share that resilience. Brains and neural nets seem to superpose memories on top of each other, all stored together in the same weights, sort of like a hologram.] superior for vision, speech, associative memory [By "associative memory", I mean noticing connections between things. One thing our brains are very good at is retrieving a pattern if we specify only a portion of the pattern.]

[It's impressive that even though a neuron needs a few milliseconds to transmit information to the next neurons downstream, we can perform very complex tasks like interpreting a visual scene in a tenth of a second. This is possible because neurons are parallel, but also because of their computation style.] [Neural nets try to emulate the parallel, associative thinking style of brains, and they are among the best techniques we have for many fuzzy problems, including some problems in vision and speech. Not coincidentally, neural nets are also inferior at many traditional computer tasks such as multiplying numbers with lots of digits or compiling source code.]

[Show figure of neurons (neurons.pdf).]

• Neuron: A cell in brain/nervous system for thinking/communication
• Action_potential or spike: An electrochemical impulse fired by a neuron to communicate w/other neurons
• Axon: The limb(s) along which the action potential propagates; "output" [Most axons branch out eventually, sometimes profusely near their ends.] [It turns out that giant squids have a very large axon they use for fast water jet propulsion. The mathematics of action potentials was first characterized in these giant squid axons, and that work won a Nobel Prize in 1963.]
• Dendrite: Smaller limbs by which neuron receives info; "input"
• Synapse: Connection from one neuron's axon to another's dendrite [Some synapses connect axons to muscles or glands.]
• Neurotransmitter: Chemical released by axon terminal to stimulate dendrite

 [When an action potential reaches an axon terminal, it causes tiny containers of neurotransmitter, called vesicles, to empty their contents into the space where the axon terminal meets another neuron's dendrite. That space is called the synaptic_cleft. The neurotransmitters bind to receptors on the dendrite and influence the next neuron's body voltage. This sounds incredibly slow, but it all happens in 1 to 5 milliseconds.]

You have about 10^11 neurons, each with about 10^4 synapses. [Maybe 10^5 synapses after you pass CS 189.]

Analogies: [between artificial neural networks and brains]

• Output of unit <-> firing rate of neuron [An action potential is "all or nothing"--they all have the same shape and size. The output of a neuron is not a fixed voltage like the output of a transistor. The output of a neuron is the frequency at which it fires. Some neurons can fire at nearly 1,000 times a second, which you might think of as a strong "1" output. Conversely, some types of neurons can go for minutes without firing. But some types of neurons never stop firing, and for those you might interpret a firing rate of 10 times per second as "0".]
• Weight of connection <-> synapse strength
• Positive weight <-> excitatory neurotransmitter (e.g. glutamine)
• Negative weight <-> inhibitory neurotransmitter (e.g. GABA, glycine) [Gamma aminobutyric acid.] [A typical neuron is either excitatory at all its axon terminals, or inhibitory at all its terminals. It can't switch from one to the other. Artificial neural nets have an advantage here.]
• Linear combo of inputs <-> summation [A neuron fires when the sum of its inputs, integrated over time, reaches a high enough voltage. However, the neuron body voltage also decays slowly with time, so if the action potentials coming in are slow enough, the neuron might not fire at all.]
• Logistic/sigmoid fn <-> firing rate saturation [A neuron can't fire more than 1,000 times a second, nor less than zero times a second. This limits its ability to be the sole determinant of whether downstream neurons fire. We accomplish the same thing with the sigmoid fn.]
• Weight change/learning <-> synaptic_plasticity Hebb's rule (1949): "Cells that fire together, wire together." [This doesn't mean that the cells have to fire at exactly the same time. But if one cell's firing tends to make another cell fire more often, their excitatory synaptic connection tends to grow stronger. There's a reverse rule for inhibitory connections. And there are ways for neurons that aren't even connected to grow connections.] [There are simple computer learning algorithms based on Hebb's rule. It can work, but it's generally not nearly as fast or effective as backpropagation.]

[Backpropagation is one part of artificial neural networks for which there is no analogy in the brain. Brains definitely do not do backpropagation.]

[Show Geoff Hinton Hebbian learning slides (hebbian.pdf).]

[But this two-layer network is not flexible enough to do digit recognition well, especially when you have multiple writers with different handwriting. You can do much better with a three-layer network and backpropagation.]

[The brain is very modular.] [Show figure of brain lobes (brain.png).

• The part of our brain we think of as most characteristically human is the cerebral cortex, the seat of self-awareness, language, and abstract thinking.

But the brain has a lot of other parts that take the load off the cortex.

• Our brain stem regulates functions like heartbeat, breathing, and sleep.
• Our cerebellum governs fine coordination of motor skills. When we talk about "muscle memory", much of that is in the cerebellum, and it saves us from having to consciously think about how to walk or talk or brush our teeth, so the cortex can focus on where to walk and what to say and checking our phone.
• Our limbic system is the center of emotion and motivation, and as such, it makes a lot of the big decisions. I sometimes think that 90% of the job of our cerebral cortex is to rationalize decisions that have alread been made by the limbic system. "Oh yeah, I made a careful, reasoned, logical decision to eat that fourth pint of ice cream."
• Our visual cortex performs a lot of processing on the input from your eyes to change it into a more useful form. Neuroscientists and computer scientists are particularly interested in the visual cortex for several reasons. Vision is an important problem for computers. The visual cortex is one of the easier parts of the brain to study in lab animals. The visual cortex is largely a feedforward network with few neurons going backward, so it's easier for us to train computers to behave like the visual cortex.]

[Although the brain has lots of specialized modules, one thing that's interesting about the cerebral cortex is that it seems to be made of general-purpose neural tissue that looks more or less the same everywhere, at least before it's trained. If you experience damage to part of the cortex early enough in life, while your brain is still growing, the functions will just relocate to a different part of the cortex, and you'll probably never notice the difference.]

[As computer scientists, our primary motivation for studying neurology is to try to get clues about how we can get computers to do tasks that humans are good at. But neurologists and psychologists have also been part of the study of neural nets from the very beginning. Their motivations are scientific: they're curious how humans think, and why we can do what we can do.]

=================== [I want to show you a few basic variations on the standard neural network I showed you last class, and how some of them change backpropagation.]

Regression: usually omit sigmoid fn from output unit(s). [If you make that change, the gradient changes too, and you have to derive it for backprop. The gradient gets simpler, so I'll leave it as an exercise.]

Classification:
- Logistic loss fn (aka _cross-entropy_) often preferred to squared error.
  L(z, y) = - sum (y  ln z  + (1 - y ) ln (1 - z ))
    ^  ^       i    i     i         i           i
    |  |
    |  true values  \ vectors
   prediction       /
- For 2 classes, use one sigmoid output; for k >= 3 classes, use _softmax_fn_.
  Let t = Wh be k-vector of linear combos in final layer.

                                t_j        d z                   d z
                               e              j                     j
  Softmax output is z (t) = --------- .    ---- = z  (1 - z )    ---- = - z  z
                     j       k    t_i      d t     j       j     d t       i  j
                            sum  e            j                     i
                            i=1                                       i != j

  [Each z_j is in the range (0, 1), and their sum is 1.]

[See my hand-drawn derivation of the backprop equations for the softmax output and logistic loss function.]

[Next I'm going to talk about a bunch of heuristics that make gradient descent faster, or make it find better local minima, or prevent it from overfitting. I suggest implementing vanilla stochastic backprop first, and experimenting with the other heuristics only after you get that working.]