Indexing a matrix with two matrixes

How to index a matrix with matrixes?

A short outline of the matrix indexing will be followed by
a=[1,2,3,4;5,6,7,8;9,10,11,12]
b=[1,2;3,3;2,3]
c=[1;2]
octave:6> a(b,c)
ans =

    1    2
    9   10
    5    6
    5    6
    9   10
    9   10

Indexing Vectors (https://de.mathworks.com/company/newsletters/articles/matrix-indexing-in-matlab.html)
Let's start with the simple case of a vector and a single subscript. The vector is:

  v = [16 5 9 4 2 11 7 14];
The subscript can be a single value:

  v(3)     % Extract the third element
  ans =
        9
Or the subscript can itself be another vector:

  v([1 5 6])      % Extract the first, fifth, and sixth elements
  ans =
        16   2   11

The crucial point here is item 1  selects item 1 of v which is 16 and
item 5 selects item 5 of v which is 2 and
item 6 selects item 6 of v which is 11.
------------------------------------------------

Let's take an a matrix with octave:
C:\Users\ars>octave -qf
octave:1> a=[1,2,3,4;5,6,7,8;9,10,11,12]
a =
    1    2    3    4
    5    6    7    8
    9   10   11   12


Let's take a b matrix:
octave:2> b=[1,2;3,4;5,6]
b =
   1   2
   3   4
   5   6


Let's take a c matrix:
octave:3> c=[1;2]
c =
   1
   2

octave:4> a(b,c)
error: A(I,J): row index out of bounds; value 6 out of bound 3
Octave does not like 6 because a has only 3 rows.

Let's take b with only one item bigger than a's row count.
octave:4> b=[1,2;3,4;2,3]
b =

   1   2
   3   4
   2   3

octave:5> a(b,c)
error: A(I,J): row index out of bounds; value 4 out of bound 3
The error repeats.  Hence, the items of b matrix must not be
bigger than the row count of a matrix.


Let's take a b matrix with all of its items less or equal to
the row count of a matrix.

octave:5> b=[1,2;3,3;2,3]
b =

   1   2
   3   3
   2   3

Let's try a(b,c) again:
 
octave:6> a(b,c)
ans =

    1    2
    9   10
    5    6
    5    6
    9   10
    9   10

It works.  Let's recall a and c matrixes and analyze this result:

a =
    1    2    3    4
    5    6    7    8
    9   10   11   12

c =
   1
   2
 
We have to refer for each of b the related row
and each of c to the realed column of a matrix.

b=1 c=1,2 -> 1 2
b=3 c=1,2 -> 9 10
b=2 c=1,2 -> 5 6
Please notice that b points to each row according to the value of item in sequence
of b 1 5 9 2 6 10 3 7 11 4 8 12
and for each row c proceeds for each column of each row with the value of items in matrix sequence
of c that is 1 2 1 2 1 2 1 2 1 2 ...
namely
b=2 c=1,2 -> 5 6
b=3 c=1,2 -> 9 10
b=3 c=1,2 -> 9 10

Just another example:

octave:7> b=[1,1;2,2;3,3]
b =

   1   1
   2   2
   3   3

Let's recal a and c matrixes:

a =
    1    2    3    4
    5    6    7    8
    9   10   11   12

c =
   1
   2
 
octave:8> a(b,c)
ans =

    1    2
    5    6
    9   10
    1    2
    5    6
    9   10

c selects first and second columns. b selects
1 2 3 1 2 3 rows.

Let's increase the rows count of b.  And observe:

octave:9> b=[1,1;2,2;3,3;1,1]
b =

   1   1
   2   2
   3   3
   1   1

octave:10> a(b,c)
ans =

    1    2
    5    6
    9   10
    1    2
    1    2
    5    6
    9   10
    1    2

Increasing the size of b (or c) does not cause an
abnormal end.  It only increases the selected items
of the output matrix.  Infact
the size of the output matrix = size(b) times size(c).
Below given examples show that, the size and shape of the
b and c matrixes a(b,c) does not cause an abend because
the a(b,c) calculation is done according to the item
sequence of b and c matrixes.


octave:11> b=[1,1,1;2,2,2;3,3,3]
b =

   1   1   1
   2   2   2
   3   3   3

octave:12> a(b,c)
ans =

    1    2
    5    6
    9   10
    1    2
    5    6
    9   10
    1    2
    5    6
    9   10

octave:13> c=[1,2,2]
c =

   1   2   2

octave:14> a(b,c)
ans =

    1    2    2
    5    6    6
    9   10   10
    1    2    2
    5    6    6
    9   10   10
    1    2    2
    5    6    6
    9   10   10

octave:15> c=[1,2;3,4]
c =

   1   2
   3   4

octave:16> a(b,c)
ans =

    1    3    2    4
    5    7    6    8
    9   11   10   12
    1    3    2    4
    5    7    6    8
    9   11   10   12
    1    3    2    4
    5    7    6    8
    9   11   10   12

Cheers.

Ali Riza SARAL
arsaral((at))yahoo.com

Otomatik Cevap Motoru - Cevap Analizi

Mehmet okula gitti.
-------------------
Mehmet nereye gitti?
Mehmet gitti mi?
Okula kim gitti?
Mehmet okula nasıl gitti?
Mehmet ne yaptı?

REF ANALİZİ
====================
0 1 Mehmet  İsim  (f_önü koşarak   %intr. *o refverbalPOS=0 ) \Nesne\   \possible Özne\ <************ÖZNE
1 2 okula  İsim->Dolaylı Tümleç -a 
2 3 koşarak  Zarf -arak  -fiilimsi  %intr.
3 4 gitti Fiil %dat.intr. *o
3 4 .crlf postPunctuation

Mehmet nereye gitti?
********************
1-Find sentences which has Mehmet as SUBJECT(ÖZNE)
2-Find sentences which has Mehmet and gitti/gitmek(from Morpholojik analysis) as Fiil.
3-Find a Dolaylı Tümleç with -e/a type (nere-ye)
4-Use this tümleç to form an answer sentence -->Mehmet OKULA gitti.

Mehmet gitti mi?
****************
1-Find sentences which has Mehmet as SUBJECT(ÖZNE)
2-Find sentences which has Mehmet and gitti/gitmek(from Morpholojik analysis) as Fiil.
3-If above items are found form answer as YES else NO.

Okula kim gitti?
****************
1-Find sentences which has gitti/gitmek(from Morpholojik analysis) as Fiil.
2-Find sentences which has okula
3-Find Özne in this sentence and ANSWER with the text of that word(s)

Mehmet okula nasıl gitti?
*************************
1-Find sentences which has Mehmet as SUBJECT(ÖZNE)
2-Find sentences which has Mehmet and gitti/gitmek(from Morpholojik analysis) as Fiil.
2-Find sentences which has okula
4- Find Zarf before or after Fiil answer with the text of that word

Mehmet ne yaptı?
****************
1-Find sentences which has Mehmet as SUBJECT(ÖZNE)
2-Limit/warn that the number may be many.
3-Extract Nesne and fiil of these sentences and form sentences such as
'Mehmet gitti'.

Otomatik Cevap Motoru - Soru Dosyası Analizi

REFERANS METİN
==============
Mehmet okula koşarak gitti.
Ayşe topu bana attı.
O matematik dersini çok çalışır.
Ahmet heyecanlı bir çocuktur.

Mehmet okula gitti.
-------------------
Mehmet nereye gitti?
Mehmet gitti mi?
Okula kim gitti?
Mehmet okula nasıl gitti?
Mehmet ne yaptı?

Ayşe topu bana attı.
--------------------
Kim topu attı?
Kim topu bana attı?
Bana topu kim attı?
Ayşe bana ne attı?
Ayşe ne yaptı?
Top ne oldu?

O matematik dersini çok çalışır.
-------------------------------
Matematik dersini kim çok çalışır?
O neyi çok çalışır?
O hangi dersi çok çalışır?
O matematik dersini ne yapar?
O ne yapar?

Ahmet heyecanlı bir çocuktur.
-----------------------------
Ahmet nasıl bir çocuktur?
Ahmet kimdir?

Otomatik Cevap Motoru - Ref Dosyası Analizi

sentence= 0<<****************************dumpSentenceAsText
Mehmet okula koşarak gitti.

sentence= 0<<----------------------------dumpstructlist p="">0 1 Mehmet  İsim  (f_önü koşarak   %intr. *o refverbalPOS=0 ) \Nesne\   \possible Özne\ <************ÖZNE
1 2 okula  İsim->Dolaylı Tümleç -a 
2 3 koşarak  Zarf -arak  -fiilimsi  %intr.
3 4 gitti Fiil %dat.intr. *o
3 4 .crlf postPunctuation


sentence= 1<<****************************dumpSentenceAsText
Ayşe topu bana attı.

sentence= 1<<----------------------------dumpstructlist p="">0 2 Ayşe topu  İsim Tamlaması -i/ı/u/ü  (f_önü attı  %tr.intr.dat. *o ref ) \Nesne\  <************NESNE
2 3 bana  Zamir->Dolaylı Tümleç -a 
3 4 attı Fiil %tr.intr.dat. *o
3 4 .crlf postPunctuation


sentence= 2<<****************************dumpSentenceAsText
O matematik dersini çok çalışır.

sentence= 2<<----------------------------dumpstructlist p="">0 1 O  Zamir   \possible Özne\ ( çalışır  %tr.intr. *o ref )<************ÖZNE
1 3 matematik dersini  Sıfat Tamlaması->Nesne -ini   (f_önü çalışır  %tr.intr. *o ref ) \Nesne\  <************NESNE
3 4 çok  Zarf
4 5 çalışır Fiil %tr.intr. *o
4 5 .crlf postPunctuation


sentence= 3<<****************************dumpSentenceAsText
Ahmet heyecanlı bir çocuktur.

sentence= 3<<----------------------------dumpstructlist p="">0 1 Ahmet  İsim  (f_önü heyecanlı bir çocuktur   %intr. *o ref ) \Nesne\    \possible Özne\ <************ÖZNE
1 4 heyecanlı bir çocuktur  Fiil İsim/Sıfat Tamlaması Türevi %intr.  -tur 
3 4 .crlfcrlfcrlfcrlfcrlf postPunctuation

BUILD SUCCESSFUL (total time: 1 second)





MORPHOLOJIK(kelime yapısı) VERİLERİ
===================================

sentence= 0<<****************************dumpSentenceAsText
Mehmet okula koşarak gitti.

sentence= 0<<----------------------------morphologic analysis="" p="">0 0   0 Mehmet
R is. ===>Mehmet mehmet
1 0   1 okula
N is. ===>okula   =okul -a ^Noun->Noun 
2 0   2 koşarak
V f. ===>koşarak   =koş -arak ^Verb->Adverb ->Adverb %intr.
3 0   3 gitti.

V f. ===>gitti   =git -ti ^DiliGeçmiş %dat.intr.


sentence= 1<<****************************dumpSentenceAsText
Ayşe topu bana attı.

sentence= 1<<----------------------------morphologic analysis="" p="">4 1   0 Ayşe
Ayşe-->ayşe
R is. ===>Ayşe ayşe
5 1   1 topu
R is. ===>topu   top -u 
6 1   2 bana
R zm. ===>bana   ben -a
7 1   3 attı.
V f. ===>attı   =at -tı ^DiliGeçmiş %tr.intr.dat.


sentence= 2<<****************************dumpSentenceAsText
O matematik dersini çok çalışır.

sentence= 2<<----------------------------morphologic analysis="" p="">8 2   0 O
O-->o
R zm. ===>O   o
9 2   1 matematik
R sf. ===>matematik
10 2   2 dersini
N is. ===>dersini   =ders -ini ^Noun->Noun 
11 2   3 çok
R zf. sf. ===>çok
12 2   4 çalışır.

V f. ===>çalışır   =çalış -ır ^IRliGenişZ %tr.intr.


sentence= 3<<****************************dumpSentenceAsText
Ahmet heyecanlı bir çocuktur.

sentence= 3<<----------------------------morphologic analysis="" p="">13 3   0 Ahmet
Ahmet-->ahmet
R is. ===>Ahmet ahmet
14 3   1 heyecanlı
R sf. ===>heyecanlı
N is. ===>heyecanlı   =heyecan -lı ^Noun->Noun/Adjective 
15 3   2 bir
R sf. ===>bir
16 3   3 çocuktur.
N is. ===>çocuktur   =çocuk -tur ^Noun->Verb %intr.

Otomatik Cevap Motoru - Ref Dosyası

Mehmet okula koşarak gitti.
Ayşe topu bana attı.
O matematik dersini çok çalışır.
Ahmet heyecanlı bir çocuktur.

mnistclassify analysis 4

CG_CLASSIFY.m
% Version 1.000
%
% Code provided by Ruslan Salakhutdinov and Geoff Hinton
%
% Permission is granted for anyone to copy, use, modify, or distribute this
% program and accompanying programs and documents for any purpose, provided
% this copyright notice is retained and prominently displayed, along with
% a note saying that the original programs are available from our
% web page.
% The programs and documents are distributed without any warranty, express or
% implied.  As the programs were written for research purposes only, they have
% not been tested to the degree that would be advisable in any important
% application.  All use of these programs is entirely at the user's own risk.


function [f, df] = CG_CLASSIFY(VV,Dim,XX,target);

l1 = Dim(1); % 784
l2 = Dim(2); % 250
l3= Dim(3); % 250
l4= Dim(4); 5 50
l5= Dim(5); % 10
N = size(XX,1); % 1000 785

% Do decomversion.
 w1 = reshape(VV(1:(l1+1)*l2),l1+1,l2); % 785 250 = 272060(1:785*250, 785,250)
 xxx = (l1+1)*l2; % 785 * 250
 w2 = reshape(VV(xxx+1:xxx+(l2+1)*l3),l2+1,l3); % 251 250
 xxx = xxx+(l2+1)*l3;
 w3 = reshape(VV(xxx+1:xxx+(l3+1)*l4),l3+1,l4); % 251 50
 xxx = xxx+(l3+1)*l4;
 w_class = reshape(VV(xxx+1:xxx+(l4+1)*l5),l4+1,l5); % 51 10


  XX = [XX ones(N,1)]; %1000 785  data+1col
  w1probs = 1./(1 + exp(-XX*w1)); w1probs = [w1probs  ones(N,1)]; % 1000 785 * 785 250 -> 1000 251
  w2probs = 1./(1 + exp(-w1probs*w2)); w2probs = [w2probs ones(N,1)]; % 1000 251 * 251 250 -> 1000 251
  w3probs = 1./(1 + exp(-w2probs*w3)); w3probs = [w3probs  ones(N,1)]; % 1000 251 * 251 50 -> 1000 51

  targetout = exp(w3probs*w_class); % 1000 51 * 51 10 = 1000 10
  targetout = targetout./repmat(sum(targetout,2),1,10); % repmat((1000 1),1,10) normalize
  f = -sum(sum( target(:,1:end).*log(targetout))) ;  % cross enthropy

IO = (targetout-target(:,1:end));  % 1000 10
Ix_class=IO;
dw_class =  w3probs'*Ix_class; % 1000 51' * 1000 10 = 51 10

Ix3 = (Ix_class*w_class').*w3probs.*(1-w3probs); % 1000 10 * 51 10' = 1000 51 .* 1000 51 = 1000 51
Ix3 = Ix3(:,1:end-1); % 1000 50
dw3 =  w2probs'*Ix3; %1000 251' * 1000 50 = 251 50

Ix2 = (Ix3*w3').*w2probs.*(1-w2probs); % 1000 50 * 251 50' .* 1000 251 = 1000 251
Ix2 = Ix2(:,1:end-1); % 1000 250
dw2 =  w1probs'*Ix2; % 1000 251' * 1000 250 = 251 250

Ix1 = (Ix2*w2').*w1probs.*(1-w1probs); % 1000 250 * 251 250' .* 1000 251 = 1000 251
Ix1 = Ix1(:,1:end-1); % 1000 250
dw1 =  XX'*Ix1; % 1000 785' * 1000 250 = 785 250

df = [dw1(:)' dw2(:)' dw3(:)' dw_class(:)']'; % 272060 1

mnistclassify analysis 3

CG_CLASSIFY_INIT.m
% Version 1.000
%
% Code provided by Ruslan Salakhutdinov and Geoff Hinton
%
% Permission is granted for anyone to copy, use, modify, or distribute this
% program and accompanying programs and documents for any purpose, provided
% this copyright notice is retained and prominently displayed, along with
% a note saying that the original programs are available from our
% web page.
% The programs and documents are distributed without any warranty, express or
% implied.  As the programs were written for research purposes only, they have
% not been tested to the degree that would be advisable in any important
% application.  All use of these programs is entirely at the user's own risk.


function [f, df] = CG_CLASSIFY_INIT(VV,Dim,w3probs,target); % 510 1, 2 1, 1000 51, 1000 10
l1 = Dim(1); % 50
l2 = Dim(2); % 10
N = size(w3probs,1); % 1000
% Do decomversion.
  w_class = reshape(VV,l1+1,l2); % 51 10
  w3probs = [w3probs  ones(N,1)];  % 1000 51

  targetout = exp(w3probs*w_class); % 1000 51 * 51 10 = 1000 10
  targetout = targetout./repmat(sum(targetout,2),1,10); % repmat((1000 1),1,10) normalize
% 
  f = -sum(sum( target(:,1:end).*log(targetout))) ; % cross enthropy
IO = (targetout-target(:,1:end)); % 1000 10
Ix_class=IO;
dw_class =  w3probs'*Ix_class; % 1000 51' * 1000 10 = 51 10

df = [dw_class(:)']'; % 510 1

%{
 debug> a=[1,2,3;4,5,6]
a =
   1   2   3
   4   5   6

debug> sum(a,2)
ans =
6
   15

debug> repmat(sum(a,2),1,10)
ans =
6    6    6    6    6    6    6    6    6    6
   15   15   15   15   15   15   15   15   15   15
%}

mnistclassify analysis 2

backpropclassify.m
% Version 1.000
%
% Code provided by Ruslan Salakhutdinov and Geoff Hinton
%
% Permission is granted for anyone to copy, use, modify, or distribute this
% program and accompanying programs and documents for any purpose, provided
% this copyright notice is retained and prominently displayed, along with
% a note saying that the original programs are available from our
% web page.
% The programs and documents are distributed without any warranty, express or
% implied.  As the programs were written for research purposes only, they have
% not been tested to the degree that would be advisable in any important
% application.  All use of these programs is entirely at the user's own risk.

% This program fine-tunes an autoencoder with backpropagation.
% Weights of the autoencoder are going to be saved in mnist_weights.mat
% and trainig and test reconstruction errors in mnist_error.mat
% You can also set maxepoch, default value is 200 as in our paper. 

maxepoch=2; %maxepoch=200;
fprintf(1,'\nTraining discriminative model on MNIST by minimizing cross entropy error. \n');
fprintf(1,'60 batches of 1000 cases each. \n');

load mnistvhclassify
load mnisthpclassify
load mnisthp2classify

makebatches;
[numcases numdims numbatches]=size(batchdata); % 100  784  600
N=numcases; % 100

%%%% PREINITIALIZE WEIGHTS OF THE DISCRIMINATIVE MODEL%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

w1=[vishid; hidrecbiases]; % 784 250 + 1 250 = 785 250
w2=[hidpen; penrecbiases]; % 250 250 + 1 250 = 251 250
w3=[hidpen2; penrecbiases2]; % 250 500 + 1 500 = 251 500
w_class = 0.1*randn(size(w3,2)+1,10); % randn(501,10)  = 501 10


%%%%%%%%%% END OF PREINITIALIZATION OF WEIGHTS  %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

l1=size(w1,1)-1; % 784
l2=size(w2,1)-1; % 250
l3=size(w3,1)-1; % 250
l4=size(w_class,1)-1; % 500
l5=10;
test_err=[];
train_err=[];


for epoch = 1:maxepoch

 %%%%%%%%%%%%%%%%%%%% COMPUTE TRAINING MISCLASSIFICATION ERROR %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 err=0;
 err_cr=0;
 counter=0;
 [numcases numdims numbatches]=size(batchdata); % 100  784  600
 N=numcases; % 100
 for batch = 1:numbatches  % 1 : 600
  data = [batchdata(:,:,batch)]; % 100  784
  target = [batchtargets(:,:,batch)]; % 100 10 600
  data = [data ones(N,1)]; % 100 785
  w1probs = 1./(1 + exp(-data*w1)); w1probs = [w1probs  ones(N,1)]; % 100 785 * 785 250 += 100 251
  w2probs = 1./(1 + exp(-w1probs*w2)); w2probs = [w2probs ones(N,1)]; % 100 251 * 251 250 += 100 251
  w3probs = 1./(1 + exp(-w2probs*w3)); w3probs = [w3probs  ones(N,1)]; % 100 251 * 251 500 += 100 501
  targetout = exp(w3probs*w_class); % 100 501 * 501 10 = 100 10
  targetout = targetout./repmat(sum(targetout,2),1,10); % 100 10 = 100 10 ./ repmat ( 100 1), 1 10) = 100 10

  [I J]=max(targetout,[],2);  % 100 1 , 100 1 = 100 1 -->I has the value J has the sequence
  [I1 J1]=max(target,[],2);  % max(100 10,[],2) 100 1
  counter=counter+length(find(J==J1));  % =6 for the first batch
  err_cr = err_cr- sum(sum( target(:,1:end).*log(targetout))) ;  %cross entrophy

 end
 train_err(epoch)=(numcases*numbatches-counter); % total number of errors for all the batches in this epoche
 train_crerr(epoch)=err_cr/numbatches; % total cross enthropy error for the complete batchdata in this epoche

 %%%%%%%%%%%%%% END OF COMPUTING TRAINING MISCLASSIFICATION ERROR %%%%%%%%%%%%%%%%%%%%%%%%%%%%%

 %%%%%%%%%%%%%%%%%%%% COMPUTE TEST MISCLASSIFICATION ERROR %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 err=0;
 err_cr=0;
 counter=0;
 [testnumcases testnumdims testnumbatches]=size(testbatchdata); % 100 784 100
 N=testnumcases; % 100
 for batch = 1:testnumbatches % 1: 100
  data = [testbatchdata(:,:,batch)]; % 100 784
  target = [testbatchtargets(:,:,batch)]; % 100 10 = (100 10 100(:,:,batch)
  data = [data ones(N,1)]; % 100 785
  w1probs = 1./(1 + exp(-data*w1)); w1probs = [w1probs  ones(N,1)]; % 100 785 *785 250 = 100 250 -> 100 251
  w2probs = 1./(1 + exp(-w1probs*w2)); w2probs = [w2probs ones(N,1)]; % 100 251 * 251 250 = 100 250 -> 100 251
  w3probs = 1./(1 + exp(-w2probs*w3)); w3probs = [w3probs  ones(N,1)]; % 100 251 * 251 50 = 100 50 -> 100 51
  targetout = exp(w3probs*w_class); % 100 51 * 51 10 = 100 10
  targetout = targetout./repmat(sum(targetout,2),1,10); % = 100 10 ./ repmat ( 100 1), 1 10) = 100 10

  [I J]=max(targetout,[],2); % 100 1 , 100 1 = 100 1 -->I has the value J has the sequence
  [I1 J1]=max(target,[],2); % max(100 10,[],2) 100 1
  counter=counter+length(find(J==J1));  % =9 for the first batch
  err_cr = err_cr- sum(sum( target(:,1:end).*log(targetout)));  %cross entrophy
 
 end
 test_err(epoch)=(testnumcases*testnumbatches-counter);  % total number of errors for all the batches in this epoche
 test_crerr(epoch)=err_cr/testnumbatches; % total cross enthropy error for the complete batchdata in this epoche
 fprintf(1,'Before epoch %d Train # misclassified: %d (from %d). Test # misclassified: %d (from %d) \t \t \n',...
            epoch,train_err(epoch),numcases*numbatches,test_err(epoch),testnumcases*testnumbatches);

 %%%%%%%%%%%%%% END OF COMPUTING TEST MISCLASSIFICATION ERROR %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

 tt=0;
 for batch = 1:numbatches/10
  fprintf(1,'epoch %d batch %d\n',epoch,batch);

  %%%%%%%%%%% COMBINE 10 MINIBATCHES INTO 1 LARGER MINIBATCH %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
  tt=tt+1;
  data=[];
  targets=[];
  for kk=1:10
   data=[data
         batchdata(:,:,(tt-1)*10+kk)]; % 1000 784
   targets=[targets
         batchtargets(:,:,(tt-1)*10+kk)];  % 1000 10
  end

 %%%%%%%%%%%%%%% PERFORM CONJUGATE GRADIENT WITH 3 LINESEARCHES %%%%%%%%%%%%%%%%%%%%%%%%%%%%%
  max_iter=3;

  if epoch<2 6="" fixed.="" holding="" irst="" nbsp="" original="" other="" p="" top-level="" update="" weights="">    N = size(data,1); % 1000 /1000 784)
    XX = [data ones(N,1)]; % 1000 785
    w1probs = 1./(1 + exp(-XX*w1)); w1probs = [w1probs  ones(N,1)]; % 1000 785 * 785 250 = 1000 250 -> 1000 251
    w2probs = 1./(1 + exp(-w1probs*w2)); w2probs = [w2probs ones(N,1)]; % 1000 251 * 251 250 = 1000 250 -> 1000 251
    w3probs = 1./(1 + exp(-w2probs*w3)); %w3probs = [w3probs  ones(N,1)]; % 1000 251 * 251 50 = 1000 50 -> 1000 51

    VV = [w_class(:)']'; % 51 10 = 510 1
    Dim = [l4; l5]; % 2 1
    [X, fX] = minimize(VV,'CG_CLASSIFY_INIT',max_iter,Dim,w3probs,targets); % 510 1, 4 1 = min(510 1,'CG_CLASS...',3, 2 1, 1000 51, 1000 10
    w_class = reshape(X,l4+1,l5); %reshape(X,51,10)= 51 10

  else
    VV = [w1(:)' w2(:)' w3(:)' w_class(:)']'; % 272060 1
    Dim = [l1; l2; l3; l4; l5]; % 5 1
    [X, fX] = minimize(VV,'CG_CLASSIFY',max_iter,Dim,data,targets); %[% 272060 1, 4 1]=mini..(272060 1,'CG_CLA..', 3, 5 1, 1000 784, 1000 10);

    w1 = reshape(X(1:(l1+1)*l2),l1+1,l2); %reshape(272060(1 : 785*250,785,250) = 785 250
    xxx = (l1+1)*l2; % 785 * 250
    w2 = reshape(X(xxx+1:xxx+(l2+1)*l3),l2+1,l3); % reshape(X(251 : 251 * 250), 251, 250); = 251 250
    xxx = xxx+(l2+1)*l3; % 785 * 250 + 251 * 250
    w3 = reshape(X(xxx+1:xxx+(l3+1)*l4),l3+1,l4); % reshape(X(xxx+1: xxx+251*50, 251,50) = 251, 50
    xxx = xxx+(l3+1)*l4; % 271550
    w_class = reshape(X(xxx+1:xxx+(l4+1)*l5),l4+1,l5); % 51 10

  end
  %%%%%%%%%%%%%%% END OF CONJUGATE GRADIENT WITH 3 LINESEARCHES %%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
 end

 save mnistclassify_weights w1 w2 w3 w_class
 save mnistclassify_error test_err test_crerr train_err train_crerr;

end

mnistclassify analysis 1

% Version 1.000
%
% Code provided by Ruslan Salakhutdinov and Geoff Hinton 
%
% Permission is granted for anyone to copy, use, modify, or distribute this
% program and accompanying programs and documents for any purpose, provided
% this copyright notice is retained and prominently displayed, along with
% a note saying that the original programs are available from our
% web page.
% The programs and documents are distributed without any warranty, express or
% implied.  As the programs were written for research purposes only, they have
% not been tested to the degree that would be advisable in any important
% application.  All use of these programs is entirely at the user's own risk.


% This program pretrains a deep autoencoder for MNIST dataset
% You can set the maximum number of epochs for pretraining each layer
% and you can set the architecture of the multilayer net.

clear all
close all

maxepoch=1; % maxepoch=50;
numhid=250; numpen=250; numpen2=50; % numhid=500; numpen=500; numpen2=2000;

fprintf(1,'Converting Raw files into Matlab format \n');
%converter;
dos('erase *.ascii');

fprintf(1,'Pretraining a deep autoencoder. \n');
fprintf(1,'The Science paper used 50 epochs. This uses %3i \n', maxepoch);

makebatches;
[numcases numdims numbatches]=size(batchdata);  % 100 784 600

fprintf(1,'Pretraining Layer 1 with RBM: %d-%d \n',numdims,numhid); % 784 250
restart=1;
rbm;
hidrecbiases=hidbiases; % 1 250
save mnistvhclassify vishid hidrecbiases visbiases; % 784 250, 1 250, 1 784

fprintf(1,'\nPretraining Layer 2 with RBM: %d-%d \n',numhid,numpen); % 250 250
batchdata=batchposhidprobs; % 100 250 600
numhid=numpen; %250
restart=1;
rbm;
hidpen=vishid; penrecbiases=hidbiases; hidgenbiases=visbiases;
save mnisthpclassify hidpen penrecbiases hidgenbiases; %hidpen 250 250, penrecbiases 1 250, hidgenbiases 1 250

fprintf(1,'\nPretraining Layer 3 with RBM: %d-%d \n',numpen,numpen2); % 250 50
batchdata=batchposhidprobs;
numhid=numpen2;
restart=1;
rbm;
hidpen2=vishid; penrecbiases2=hidbiases; hidgenbiases2=visbiases;
save mnisthp2classify hidpen2 penrecbiases2 hidgenbiases2; %hidpen2 250 50, penrecbiases2 1 50, hidgenbiases2 1 250

backpropclassify;