《图文并茂的 CNN 介绍 PPT.ppt》由会员分享，可在线阅读，更多相关《图文并茂的 CNN 介绍 PPT.ppt（47页珍藏版）》请在三一办公上搜索。
1、Convolution Neural NetworkCNN,A tutorialKH Wong,Convolution Neural Network CNN ver.4.11a,1,Introduction,Very Popular:Toolboxes:cuda-convnet and caffe(user friendlier)A high performance Classifier(multi-class)Successful in handwritten optical character OCR recognition,speech recognition,image noise r
2、emoval etc.Easy to implementationSlow in learningFast in classification,Convolution Neural Network CNN ver.4.11a,2,Overview of this note,Part 1:Fully connected Back Propagation Neural Networks(BPNN)Part 1A:feed forward processingPart 1A:feed backward processingPart 2:Convolution neural networks(CNN)
3、Part 2A:feed forward of CNNPart 2B:feed backward of CNN,Convolution Neural Network CNN ver.4.11a,3,Part 1,Fully Connected Back Propagation(BP)neural net,Convolution Neural Network CNN ver.4.11a,4,TheoryFully connected Back Propagation Neural Net(BPNN),Use many samples to train the weights,so it can
4、be used to classify an unknown input into different classesWill explainHow to use it after training:forward passHow to train it:how to train the weights and biases(using forward and backward passes),Convolution Neural Network CNN ver.4.11a,5,Training,How to train it:how to train the weights(W)and bi
5、ases(b)(use forward,backward passes)Initialize W and b randomlyIter=1:all_epocks(each is called an epcok)Forward pass for each output neuron:Use training samples:Xclass_t:feed forward to find y.Err=error_function(y-t)Backward pass:Find W and b to reduce Err.Wnew=Wold+W;bnew=bold+b,Convolution Neural
6、 Network CNN ver.4.11a,6,Part 1A,Forward pass of Back Propagation Neural Net(BPNN)Recall:Forward pass for each output neuron:-Use training samples:Xclass_t:feed forward to find y.-Err=error_function(y-t),Convolution Neural Network CNN ver.4.11a,7,Feed forward of Back Propagation Neural Net(BPNN),In
7、side each neuron:,Convolution Neural Network CNN ver.4.11a,8,Output neurons,Inputs,Sigmod function f(u)and its derivative f(u),Convolution Neural Network CNN ver.4.11a,9,http:/,A single neuron,The neural net can have many layersIn between any neighboring 2 layers,a set of neurons can be found,Convol
8、ution Neural Network CNN ver.4.11a,10,Each Neuron,BPNN Forward pass,Forward pass is to find output when an input is given.For example:Assume we have used N=60,000 images to train a network to recognize c=10 numerals.When an unknown image is input,the output neuron corresponds to the correct answer w
9、ill give the highest output level.,Convolution Neural Network CNN ver.4.11a,11,10 output neurons for 0,1,2,.,9,Inputimage,The criteria to train a network,Is based on the overall error function,Convolution Neural Network CNN ver.4.11a,12,Structure of a BP neural network,Convolution Neural Network CNN
10、 ver.4.11a,13,Input layer,output layer,Architecture(exercise:write formulas for A1(i=4)and A2(k=3),Convolution Neural Network CNN ver.4.11a,Input:P=9x1Indexed by j,Hidden layer=5 neurons,indexed by iW1=9x5b1=5x1,W1(j=1,i=1),W1(j=2,i=1),W1(j=9,i=1),P(j=1)P(j=2)P(j=3):P(j=9),A1(i=1),P(j=1),P(j=2),P(j=
11、9),Neuron i=1Bias=b1(i=1),W2(i=1,k=1),W2(i=2,k=1),W2(i=5,k=1),A2(k=2),A1,A2,A5,Neuron k=1Bias=b2(k=1),W1(j=1,i=1),W1(j=2,i=1),W1(j=9,i=5),W1(j=3,i=4),A1(i=5),A1(i=1),Output neurons=3 neurons,indexed by kW2=5x3b2=3x1,W2(i=5,k=3),W2(i=1,k=1),W2(i=2,k=2),W2(i=2,k=1),A1(i=2),14,Answer(exercise:write val
12、ues for A1(i=4)and A2(k=3),P=0.7656 0.7344 0.9609 0.9961 0.9141 0.9063 0.0977 0.0938 0.0859W1=0.2112 0.1540-0.0687-0.0289 0.0720-0.1666 0.2938-0.0169-0.1127-b1=0.1441%Find A1(i=4)A1_i_is_4=1/(1+exp-(W1*P+b1)=0.49,Convolution Neural Network CNN ver.4.11a,15,Numerical example for the forward path,Feed
13、 forwardGive numbers of x,w b etc,Convolution Neural Network CNN ver.4.11a,16,Example:a simple BPNN,Number of classes(no.of output neurons)=3Input 9 pixels:each input is a 3x3 imageTraining samples=3 for each classNumber of hidden layers=1Number of neurons in the hidden layer=5,Convolution Neural Ne
14、twork CNN ver.4.11a,17,Architecture of the example,Convolution Neural Network CNN ver.4.11a,18,Input Layer9x1 pixels,output Layer 3x1,Part 1B,Backward pass of Back Propagation Neural Net(BPNN),Convolution Neural Network CNN ver.4.11a,19,feedback,Convolution Neural Network CNN ver.4.11a,20,Feedforwar
15、d,Feedbackward,Convolution Neural Network CNN ver.4.11a,21,derivation,derivation,Convolution Neural Network CNN ver.4.11a,22,Numerical example for the feed back pass,Convolution Neural Network CNN ver.4.11a,23,Procedure,From the last layer(output),find dt-yFind d,then find w of the whole networkFind
16、 iterative(forward-back forward pass)to generate a new set of W,until dW is smallTakes a long time,Convolution Neural Network CNN ver.4.11a,24,Part 2Convolution Neural Networks,Part 2AFeed forward part of cnnff(),Convolution Neural Network CNN ver.4.11a,25,Matlab example,An example optical chartered
17、 recognition OCR,Example test_example_CNN.m in Based on a data base(mnist_uint8,from)60,000 training examples(28x28 pixels each)10,000 testing samples(a different dat.2set)After training,given an unknown image,it will tell whether it is 0,or 1,.,9 etc.Recognition rate 11%use 1 epoch(training 200seco
18、nds)Recognition rate 1.2%use 100 epochs(hours of training),Convolution Neural Network CNN ver.4.11a,26,Overview ofTest_example_CNN.m,Read data basePart I:cnnsetup.mLayer 1:input layer(do nothing)Layer 2 convolution(conv.)Layer,output maps=6,kernel size=5x5Layer 3 sub-sample(subs.)Layer,scale=2Layer
19、4 conv.Layer,output maps=12,kernel size=5x5Layer 5 subs.Layer(output layer),scale=2Part 2:cnntrain.m%train wedihgts using 60,000 samplescnnff()%CNN feed forwardcnndb()%CNN feed back to train weighted in kernelscnnapplygrads()%update weightscnntest.m%test the system using 10000 samples and show error
20、 rate,Convolution Neural Network CNN ver.4.11a,27,Architecture,Convolution Neural Network CNN ver.4.11a,28,Each output neuron corresponds to a character(0,1,2,.,9 etc.),Layer 1:Image Input1x28x28,Layer 12:6 conv.Maps(C)InputMaps=6OutputMaps=6Fan_in=52=25 Fan_out=6x52=150,Layer 23:6 sub-sample Map(S)
21、InputMaps=6OutputMaps=12,Layer 34:12 conv.Maps(C)InputMaps=6OutputMaps=12Fan_in=6x52=150Fan_out=12x52=300,Layer 45:12 sub-sample Map(S)InputMaps=12OutputMaps=12,Layer 1:One input(I),Layer 5:12x4x4,10,Kernel=5x5,2x2,Kernel=5x5,Subs,Subs,Layer 4:12x8x8,Layer 3:6x12x12,Layer 2:6x24x24,Conv.,I=inputC=Co
22、nv.=convolutionS=Subs=sub sampling,Conv.,2x2,Cnnff.mconvolution neural networks feed forward,This is the feed forward partAssume all the weights are initialized or calculated,we show how to get the output from inputs.,Convolution Neural Network CNN ver.4.11a,29,Layer 12:,Convolute layer 1 with diffe
23、rent kernels(map_index1=1,2,.,6)and produce 6 output mapsInputs:input layer 1,a 28x28 image6 different kernels:k(1),.,k(6),each k is 5x5,K are dendrites of neurons Output:6 output maps each 24x24AlgorithmFor(map_index=1:6)layer_2(map_index)=I*k(map_index)validDiscussionValid means only consider over
24、lapped areas,so if layer 1 is 28x28,kernel is 5x5 each,each output map is 24x24 In Matlab use convn(I,k,valid)Example:I=rand(28,28)k=rand(5,5)size(convn(I,k,valid)ans 24 24,Convolution Neural Network CNN ver.4.11a,30,Layer 1:Image Input(i)1x28x28,Layer 12:6 conv.Maps(C)InputMaps=6OutputMaps=6Fan_in=
25、52=25 Fan_out=6x52=150,Layer 1:One input(I),Kernel=5x5,2x2,Layer 2(c):6x24x24,Conv.*K(6),I=inputC=Conv.=convolutionS=Subs=sub sampling,i,j,Map_index=1 2:6,Conv.*K(1),Layer 23:,Convolution Neural Network CNN ver.4.11a,31,Layer 23:6 sub-sample Map(S)InputMaps=6OutputMaps=12,2x2,Subs,Layer 3(s):6x12x12
26、,Layer 2(c):6x24x24,Sub-sample layer 2 to layer 3Inputs:6 maps of layer 2,each is 24x24Output:6 maps of layer 3,each is 12 x12AlgorithmFor(map_index=1:6)For each input map,calculate the average of 2x2 pixels and the result is saved in output maps.Hence resolution is reduced from 24x24 to 12x12Discus
27、sion,Map_index=1 2:6,Layer 34:,Convolution Neural Network CNN ver.4.11a,32,Layer 34:12 conv.Maps(C)InputMaps=6OutputMaps=12Fan_in=6x52=150Fan_out=12x52=300,Kernel=5x5,Layer 4(c):12x8x8,Layer3 L3(s):6x12x12,Conv.layer 3 with kernels to produce layer 4Inputs:6 maps of layer3(L3i=1:6),each is 12x12Kern
28、el set:totally 6x12 kernels,each is 5x5,i.e.Ki=1:6j=1:12,each Kij is 5x512 biasj=1:12 in this layer,each is a scalarOutput:12 maps of layer4(L4j=1:12),each is 8x8Algorithmfor(j=1:12)for(i=1:6)clear z,i.e.z=0;z=z+covn(L3i,kij,valid)%z is 8x8 L4j=sigm(z+baisj)%L4j is 8x8 function X=sigm(P)X=1./(1+exp(
29、-P);EndDiscussionNormalization?,Index=i=1:6,Index=j=1:12,:,net.layersl.aj,Layer 45,Convolution Neural Network CNN ver.4.11a,33,Layer 45:12 sub-sample Map(S)InputMaps=12OutputMaps=12,Layer 5:12x4x4,10,Subs,Layer 4:12x8x8,2x2,Subsample layer 4 to layer 5Inputs:12 maps of layer4(L4i=1:12),each is 12x8x
30、8Output:12 maps of layer5(L5j=1:12),each is 4x4AlgorithmSub sample each 2x2 pixel window in L4 to a pixel in L5DiscussionNormalization?,Layer 5output,Convolution Neural Network CNN ver.4.11a,34,Each output neuron corresponds to a character(0,1,2,.,9 etc.)net.om=1:10,Layer 45:12 sub-sample Map(S)Inpu
31、tMaps=12OutputMaps=12,Layer 5(L5j=1:12:12x4x4=192Totally 192 pixels,10,Subsample layer 4 to layer 5Inputs:12 maps of layer5(L5i=1:12),each is 4x4,so L5 has 192 pixels in totalOutput layer weights:Net.ffWm=1:10p=1:192,total number of weights is 192Output:10 output neurons(net.om=1:10)AlgorithmFor m=1
32、:10%each output neuronclear net.fvnet.fv=Net.ffWmall 192 weight.*L5(all corresponding 192 pixels)net.om=sign(net.fv+bias)Discussion,:,Totally192 weights for each output neuron,Same for each output neuron,Part 2B,Back propagation partcnnbp()cnnapplyweight(),Convolution Neural Network CNN ver.4.11a,35
33、,cnnbp()overview(output back to layer 5,Convolution Neural Network CNN ver.4.11a,36,Ref:See,Layer 5 to 4,Expand 1x1 to 2x2,Convolution Neural Network CNN ver.4.11a,37,Layer 4 to 3,Rotated convolutionFind dE/dx at layer 3,Convolution Neural Network CNN ver.4.11a,38,Layer 3 to 2,Expand 1x1 to 2x2,Conv
34、olution Neural Network CNN ver.4.11a,39,Calculate gradient,From later 2 to layer 3From later 3 to layer 4Net.ffWNet.ffb found,Convolution Neural Network CNN ver.4.11a,40,Details of calc gradients,%part%reshape feature vector deltas into output map styleL4(c)run expand onlyL3(s)run conv(rot180,fill),
35、found dL2(c)run expand only%Part%calc gradientsL2(c)run conv(valid),found dk and dbL3(s)not run hereL4(c)run conv(valid),found dk and dbDone,found these for the output layer L5:net.dffW=net.od*(net.fv)/size(net.od,2);net.dffb=mean(net.od,2);,Convolution Neural Network CNN ver.4.11a,41,cnnapplygrads(
36、net,opts),For the convolution layers,L2,L4From k and dk find new k(weights)From b and db find new b(bias)For the output layer L5net.ffW=net.ffW-opts.alpha*net.dffW;net.ffb=net.ffb-opts.alpha*net.dffb;opts.alpha is to adjust learning rate,Convolution Neural Network CNN ver.4.11a,42,appendix,Convoluti
37、on Neural Network CNN ver.4.11a,43,Architecture,Convolution Neural Network CNN ver.4.11a,44,Each output neuron corresponds to a character(0,1,2,.,9 etc.),Layer 1:Image Input1x28x28,Layer 12:6 conv.Maps(C)InputMaps=6OutputMaps=6Fan_in=52=25 Fan_out=6x52=150,Layer 23:6 sub-sample Map(S)InputMaps=6Outp
38、utMaps=12,Layer 34:12 conv.Maps(C)InputMaps=6OutputMaps=12Fan_in=6x52=150Fan_out=12x52=300,Layer 45:12 sub-sample Map(S)InputMaps=12OutputMaps=12,Layer 1:One input(I),Layer 5:12x4x4,10,Kernel=5x5,2x2,Kernel=5x5,Subs,Subs,Layer 4:12x8x8,Layer 3:6x12x12,Layer 2:6x24x24,Conv.,I=inputC=Conv.=convolution
39、S=Subs=sub sampling,Conv.,2x2,j,i,u,v,A single neuron,The neural net has many layersIn between any neighboring 2 layers,a set of neurons can be found,Convolution Neural Network CNN ver.4.11a,45,Each Neuron,Derivation,dE/dW=changes at layer l+1 by changes in layer lAt output layer LdE/db=dE=f(wx+b)dE/db=d,Convolution Neural Network CNN ver.4.11a,46,References,WikiMatlab programsNeural Network for pattern recognition-Tutorial CNN Matlab example,Convolution Neural Network CNN ver.4.11a,47,
