lecture 2

Question 1

Neural networks, connectionism, parallel distributed processing

Answer 1

Based on abstract view of the neuron

The connections determine the function of the network

Connections can be formed by learning and do not need to be programmed

Question 2

Logic gates

Answer 2

Computers have electronic elements that implement ‘logic gates; and with these you can build and run programs

Question 3

McCullock-Pitts neuron - assumptions

Answer 3

1. The activity of the neuron is an ‘all or none’ process
   a. Activation is either 0 or 1
2. A certain fixed number of synapses must be excited within the period of latent addition in order to excite a neuron at any time
3. The only significant delay within the NS is a synaptic delay
4. The activity of any inhibitory synapse absolutely prevents excitation of the neuron at any time.
   a. This is currently not the case anymore. Now its weighted
5. The structure of the net does not change with time

Question 4

McCullock-Pitts neuron - 1 major thing the didn’t take into account

Answer 4

Real neurons are very noisy

 So computation in the brain is fault tolerant, which doesn’t work in a computational model (that would give an error)

 i.e., the brain does not work like a turing machine

 so neural networks abstract strongly from the details of real neurons

Question 5

neural networks abstract strongly from the details of real neurons

in what way do they differ

Answer 5

1. Conductivity delays are neglected
2. An output signal is discrete or a real-valued number
3. Net input is calculated as the weighed sum of the input sigals
4. Net input is transformed into an output signal via a simple function

Question 6

Error-correcting learning

Answer 6

form of supervised learning

Question 7

Perceptron

Answer 7

Original perceptron had only 2 layers (input and output layer)

Question 8

Limitations of the perceptron

Answer 8

1. Only binary values
   > remedied by the delta-rule
2. Only 2 layers

Question 9

Perceptron convergence theorem

Answer 9

If a pattern set can be represented by a two-layer perceptron, the perceptron learning rule will always be able to find some correct weights

o So if it can, it will.
o Does not say anything about how fast. Could be a slow process. But it will find it in the end.

Question 10

needed for error backpropagation

Answer 10

1. Algorithm to train perceptrons with more than 2 layers
2. Preferably also one that used continuous and nonlinear activation rules

Question 11

Characteristics of backpropagation

Answer 11

1. Any number of layers
2. only feedforward, no cycles
3. uses continuous nodes
   > activation between 0 and 1
4. intial weights are random
5. total error never increases
   > gradient descent in error space
   > so it goes down a little bit or stays the same

Question 12

backprop trick

Answer 12

 We have a node h in the hidden layer

 We go to the error signal on the output layer that is calculated for each node
o Error = the difference between the target and the spontaneous output

 We take all those errors in the output layer and add them up. This is the error we have for our hidden layer
o Can be positive and negative

 Not biologically plausible because axons only work in 1 direction.

Question 13

Backpropagation algorithm in rules

Answer 13

1. weight change = small constant x error x input activation
2. for an output node, the error is
   o error = (targed activation – output activation) x output activation x (1-output activation)
   o you add this to do gradient descent
3. for a hidden node, the error is
   a. error = weighted sum of to-node errors x hidden activation x (1 – hidden activation)
4. weight change and momentum.
   a. weight change = small constant x error x input activation + momentum constant x old weight change

Question 13

Disadvantages backprop

Answer 13

1. learning is slow
2. new learning will rapidly overwrite old representations unless they are interleaved with the new patterns
3. this makes it hard to keep networks up to date with new information
4. this also makes in very implausible as a psychological model of human memory

Question 14

advantages backprop

Answer 14

1. easy to use
   a. few parameters
   b. algorithm is easy to implement
2. can be applied to a wide range of data
3. very popular
4. paved the way for deep learning

Question 15

gradient descent in error space (= find the steepest slope down), but

Answer 15

i. does not guarantee high performance

ii. does not prevent getting stuck in local minima (unlike perceptron that will find the solution if there is one)

iii. the learning rule is complicated and tends to slow down