To use tips

These are the few things which are important in ML. I too sometimes forget these.. :P

• Use version control.
• Separate code from data.
• Separate input data, working data and output data.
• Modify input data with care.
• Save everything to disk frequently.
• Separate options from parameters.
• Do not use global variables.
• Record the options used to generate each run of the algorithm.

% store the results
serialise(options, 'options.dat', options.working_path);
serialise(parameters, 'parameters.dat', options.working_path); 

OR

serialise(parameters, 'parameters.dat'], ...
              [options.working_path '_' options_configuration.name]);

• Make it easy to sweep options.

• Make it easy to execute only portions of the code.

run_experiment('dataset_1_options', '|preprocess_data|initialise_model|train_model|'); 

• Use checkpointing.

% set the options
options = ...

% load the data
data = ...

if saved_state_exists(options)

    % load from disk
    [parameters, state] = deserialize_latest_params_state(options.working_path);

    % command line output
    disp(['Starting from iteration ' state.iteration]);

else

    % initialize
    parameters = init_parameters();
    state = init_state();

end

% learn the parameters
parameters = train_model(options, data, parameters, state);

• Write demos and tests.

From  hunch.net which further has its root to link

Why combination of L1 and L2 norm induces sparsity

There is a nice blog on this HERE.

Compiling mex files on 64bit Linux(UBUNTU) using MATLAB 2012

I could not setup the compiler for myself to create mex files. So I searched the internet, going through many blogs and posts. Finally I got a tutorial and 2 posts which I have almost copied shamelessly(maybe for my own reference.. :) ). The source link is at the end of the post.

This is a simple guide to compiling and running mex files from MATLAB R2012a on Ubuntu 12.04 64bit


I compiled my sweet hello.c file(You can get this hello.c in this text DOWN)
compile hello.c using the command in MATLAB:

>> mex hello.c

I had no idea what the problem was. But I was getting a warning that
Warning: You are using gcc version "4.6.3-1ubuntu5)".  The version currently supported with MEX is "4.4.6".


Although I searched the internet and few blogs told I need not downgrade to 4.4.6.(But still I installed gcc-4.4 on top of gcc-4.6)

run this command in terminal: 
sudo apt-get install gcc-4.4

The gcc version compatible with MATLAB R2012a is gcc-4.4 while the version pre-installed on Ubuntu 12.04 is gcc-4.6

Just follow these steps:

1. Open terminal and type
sudo gedit /usr/local/MATLAB/R2012a/bin/mexopts.sh
(OR where you installed your MATLAB)

2. Change 'gcc' to 'gcc-4.4' ,'g++' to 'g++-4.4' , 'gfortran' to 'gfortran-4.4' at all instances of  CC = 'gcc' , CXX = 'g++' and FC = 'gfortran'.
Save the file and exit.

3. Open MATLAB and type:
mex -setup (in the Command line).

MATLAB will show the following:

The options files available for mex are:


  1: /usr/local/MATLAB/R2012a/bin/mexopts.sh :
      Template Options file for building gcc MEX-files


  0: Exit with no changes

Enter the number of the compiler (0-1):

Select 1.

The setup is complete. Now time to test it.

4.  

If you have your code its fine. 

or this is a sample code(hello.c :P)

//SAMPLE1

#include <mex.h>

void mexFunction(int nlhs, mxArray *plhs[], int nrhs, const mxArray *prhs[])
{
  mexPrintf("Hello World!\n");
} 

//SAMPLE2 

#include "mex.h"
#include "stdio.h"
#include "matrix.h"

void mexFunction(int nlhs,mxArray *plhs[],int nrhs, const mxArray *prhs[])
{

mxArray *xdata;
double *xValues;
int i,j,r,c;
double avg;

xdata = prhs[0];
xValues = mxGetPr(xdata);

r = mxGetN(xdata);
c = mxGetM(xdata);
for(i=0;i<r;i++)
{
    avg=0.0;
    for(j=0;j<c;j++)
        avg += *(xValues++);
   
    avg = avg/c;
    printf("avg of column %d is %f \n",i+1, avg);
}

}

A fast short tutorial on creating mex files:
pdf 
epub (looks good.. I mean soothing to eyes.. :P)

Save the text file as "hello.c"

Again compile your sweet "hello.c" in MATLAB command line:
>> mex hello.c

Now I got the following warning.

/usr/bin/ld: cannot find -lstdc++

Shit got real!!! 

---------------------------------------------------------------------------------------------------------------------


Don't Panic!!!

To fix this you need to find your mexopts.sh file and change the line

 CLIBS="$CLIBS -lstdc++"

 TO  (for 64bit)

 CLIBS="$CLIBS -L/usr/local/MATLAB/R2012a/sys/os/glnxa64 -lstdc++"

 OR  (for 32bit)
 CLIBS="$CLIBS -L/usr/local/MATLAB/R2012a/sys/os/glnx86 -lstdc++"
 Please search the path and see that it exists.

obviously, you’ll need to change /usr/local/MATLAB/ to wherever you actually installed MATLAB.

Your next step is to do the following in a bash prompt

ln -s /usr/local/MATLAB/R2012a/sys/os/glnxa64/libstdc++.so.6 

/usr/local/MATLAB/R2012a/sys/os/glnxa64/libstdc++.so

(Please see that the path exists. Maybe /usr/local/MATLAB/R2012a/sys/os/glnx86) 

again – substituting wherever you installed MATLAB for /usr/local/MATLAB/

 

5. Now change your MATLAB folder to the folder you saved your text file in and type this in the command window:  

>>mex hello.c

It compiles. :)

6. The file "hello.mexglx" or "hello.mexa64" depending on your OS (32 bit / 64 bit) will show up in the same directory. 

7. Now run it from MATLAB.

(for SAMPLE1)
>> hello

(for SAMPLE2)
>> x = [1 2; 3 4];
>> hello(x);

 

References: 
link1
link2
link3


Also came to know a few things about Matlab (In next post.. :) )

Unsupervised Feature Learning and Deep Learning Resources

Unsupervised feature learning and deep learning has been fascinating to me recently and here are some interesting links and tutorials. 

First is Andrew Ng UFLDL tutorial which has some explanation and intuition. It also has starter code where we need to complete only the objective code. 

http://ufldl.stanford.edu/wiki/index.php/UFLDL_Tutorial  by Andrew Ng
 

Reducing the Dimensionality of Data with Neural Networks(pdf)

Self-taught learning(pdf)

An interesting reading about applications of Deep Learning at Google
link1
link2
link3

Google Research ICML paper(pdf)

Learning Deep Architectures for AI

Transforming Autoencoder(pdf)

Extracting and Composing Robust Features with Denoising Autoencoders(pdf)

(pdf   8 pages)

Greedy Layer-Wise Training of Deep Networks(pdf)

ML Quick start guide

I was seeing Andrew Ng's ML  lectures and I was half way.
Then I was surprised when I saw a link where there were some prerequisites suggested by someone which I never thought were prerequisites for Basic ML.
All were good links(the prerequisite list were the ones I knew) which i wanted to complete after ANg's lecture.
I would like to post them here for people who want to give time for ML and for my own reference.  

I myself have not completed the prerequisites but I am sure that all of them are good(everyone suggests the same).

Quick start guide(Video Links)

People generally ignore this but knowing about Algorithms of CS is also important.(CORMEN/Skiena/Kleinberg)

The following knowledge is prerequisite to make any sense out of Machine learning

    Linear Algebra by Gilbert Strang: http://ocw.mit.edu/courses/mathematics/18-06-linear-algebra-spring-2010/video-lectures/
    Convex Optimization by Boyd http://see.stanford.edu/see/courseinfo.aspx?coll=2db7ced4-39d1-4fdb-90e8-364129597c87
    Probability and statistics for ML: http://videolectures.net/bootcamp07_keller_bss/
    Some mathematical tools for ML: http://videolectures.net/mlss03_burges_smtml/ Video+Audio Very bad quality
    Probability primer (measure theory and probability theory) : http://www.youtube.com/playlist?list=PL17567A1A3F5DB5E4&feature=plcp

Once the prerequisites are complete,  the following are good series of lectures on Machine Learning.
Basic ML:

    Andrew Ng’s Video Lectures(CS229) : http://see.stanford.edu/see/courseinfo.aspx?coll=348ca38a-3a6d-4052-937d-cb017338d7b1
    Andrew Ng’s online course offering: http://www.ml-class.org
    Tom Mitchell’s video lectures(10-701) : http://www.cs.cmu.edu/~tom/10701_sp11/lectures.shtml
    Mathematicalmonk’s videos: http://www.youtube.com/playlist?list=PLD0F06AA0D2E8FFBA&feature=plpp
    Learning from data by CalTech:     http://work.caltech.edu/telecourse.html


Advanced ML:
    Probabilistic graphical models by Daphne Koller(Stanford)
    http://www.pgm-class.org/

   
    SVMs and kernel methods , Scholkopf: http://videolectures.net/mlss03_scholkopf_lk/
    basics for Support Vector Machines and related Kernel methods. Video+Audio Very bad quality
    Kernel methods and Support Vector Machines, Smola: http://videolectures.net/mlss08au_smola_ksvm/
    Introduction of the main ideas of statistical learning theory, Support Vector Machines, Kernel Feature Spaces, An overview of the applications of Kernel Methods.
    Easily one of the best talks on SVM. Almost like a run-down tutorial. http://videolectures.net/mlss06tw_lin_svm/
    Introduction to Learning Theory, Olivier Bousquet.  http://videolectures.net/mlss06au_bousquet_ilt/
    This tutorial focuses on the “larger picture” than on mathematical proofs, it is not restricted to statistical learning theory however. 5 lectures.
    Statistical Learning Theory, Olivier Bousquet, http://videolectures.net/mlss07_bousquet_slt/
    This course gives a detailed introduction to Learning Theory with a focus on the Classification problem.
    Statistical Learning Theory, John-Shawe Taylor, University of London.  7 lectures.  http://videolectures.net/mlss04_taylor_slt/
    Advanced Statistical Learning Theory, Oliver Bousquet. 3 Lectures. http://videolectures.net/mlss04_bousquet_aslt/

Most of the above links have been filtered from http://onionesquereality.wordpress.com/2008/08/31/demystifying-support-vector-machines-for-beginners/
 Important Links:

    Channel for probability primer and Machine learning . : http://www.youtube.com/user/mathematicalmonk#grid/user/D0F06AA0D2E8FFBA [VIDEO]
    A comprehensive blog comprising of best resources for ML : http://onionesquereality.wordpress.com/2008/08/31/demystifying-support-vector-machines-for-beginners/ [links]
    Another great blog for ML http://www.quora.com/Machine-Learning/What-are-some-good-resources-for-learning-about-machine-learning-Why [links]
    Lectures 21-28 by Gilbert Strang, linear algebra way of optimization.  http://academicearth.org/courses/mathematical-methods-for-engineers-ii

Good Reads

I came across an awesome  posts and just posted it here for my repository.
Best todo list ever..
http://www.paulgraham.com/todo.html


Teach Yourself Programming in Ten Years
Why is everyone in such a rush?
by
Peter Norvig

http://norvig.com/21-days.html

How to become a hacker? eric raymond

http://www.catb.org/~esr/faqs/hacker-howto.html

by paul graham 

http://www.paulgraham.com/college.html

http://www.paulgraham.com/gh.html 

http://www.paulgraham.com

python paradox

http://www.paulgraham.com/pypar.html

https://michaelochurch.wordpress.com/2012/07/27/six-languages-to-master/

http://kangalex.com/post/31729006624/45-life-lessons-written-by-a-90-year-old

Using multiple functions in single file(globally)

Declaration: 

function funs = makefuns
  funs.fun1=@fun1;
  funs.fun2=@fun2;
end

function y=fun1(x)
  ...
end

function z=fun2
  ...
end

Calls: 

myfuns = makefuns;
myfuns.fun1(x)    
myfuns.fun2() 

see also for object oriented programming in matlab: 

http://yagtom.googlecode.com/svn/trunk/html/objectOriented.html