Neural Networks in Python
Before You Begin
We've been using the Keras API with the TensorFlow backend for our simulations. Our code often also uses the numPy and SciKit libraries because I often steal code examples that happen to have been coded using those libraries. When you get started, you will want to make sure that all the requisite libraries are installed. Pip (the Python package installer) has a function called freeze
that will list all your installed libraries. This list can be dumped to a text file and used to share the list of Python packages you'll need to run any of the code we've developed:
pip freeze > requirements.txt
Now any lab member can use requirements.txt to ensure they have the requisite Python packages installed:
pip install -r requirements.txt
I've got the current list of dependencies on ubfs/Scripts/Python/requirements.txt
A Simple Feedforward Classifier Network
Keras has defined network templates, but the only one that I've seen well-documented is the 'forward model. Fortunately, the forward model is well-suited to the classification task: it takes a set of input values, feeds the values forward over a set of layers, and checks the output values in a set of binary classification units. If the output decision is strictly binary (A vs B), then one output unit will suffice (since it's a zero-sum decision). If 3+ output categories are possible, then there needs to be an output unit for each possible category.
import numpy as np import keras import time import scipy.io as sio from keras.layers import Input,Dense,Dropout from keras.models import Model, Sequential, load_model from keras.regularizers import l1 from keras.optimizers import Adadelta from sklearn.model_selection import StratifiedKFold as SKF from sklearn.model_selection import ShuffleSplit as SS import read_activations as reader seed=int(time.time()) np.random.seed(seed) trainfile='av_hi.csv' traindata=np.genfromtxt(trainfile, delimiter=',') testdata=(traindata) x_train=traindata[:,0:-1] y_cat_code=testdata[:,-1] #this next line converts the category codes of 1,2,3 into 3 separate output units so that #a code of 1 maps to {1,0,0}, 2 maps to {0,1,0} and 3 maps to {0,0,1} y_cat_train=keras.utils.to_categorical(y_cat_code, num_classes=3) encoding_dim1=96 encoding_dim2=32 input_trials, inodes = x_train.shape foldnumber=0 fp=open('perflog.txt', 'w') hfp=open('cat_traininghist.txt', 'w') kfold = SKF(n_splits=8, shuffle=True, random_state=seed) for train, test in kfold.split(x_train, y_cat_code): foldnumber=foldnumber+1 ada=keras.optimizers.Adadelta(lr=1, rho=0.9, decay=1e-6) #input layer: a series of 1000 floats from 0.0 to 1.0 main_input=Input(shape=(inodes,), dtype='float32', name='input') #first dropout layer, initially, lets randomly drop 20% of values dropout_1=Dropout(0.2, input_shape=(inodes,), name='dense_1_drop')(main_input) #noise layer to mitigate overfitting noise_1=keras.layers.noise.GaussianNoise(0.05, input_shape=(inodes,), name='dense_1_noise')(dropout_1) #first hidden layer dense_1=Dense(encoding_dim1, activation='relu', input_dim=inodes, name='dense_1', kernel_regularizer=l1(0.00001))(noise_1) #second hidden layer dense_2=Dense(encoding_dim2, activation='relu', name='dense_2')(dense_1) #category output layer cat_output=Dense(3, activation='softmax', name='cat_output')(dense_2) #assemble the model model=Model(inputs=[main_input], outputs=[cat_output]) #compile the model model.compile(optimizer='adadelta', loss=['categorical_crossentropy'], metrics=['accuracy'], loss_weights=[1]) X_train=x_train[train] X_test=x_train[test] Y_cat_train=y_cat_train[train] Y_cat_test=y_cat_train[test] xrows, xcols = X_test.shape history=model.fit(X_train, [Y_cat_train], validation_split=0.2, epochs=64, batch_size=32, shuffle=True, verbose=0) trainscore = model.evaluate(X_test, [Y_cat_test], verbose=0) outstr='%.2f\n' % trainscore[1] print outstr fp.write(outstr) fname='model_' + format(foldnumber, '02') + '.h5' model.save(fname) #save the trained model #training history vals=history.history['val_acc'] strlist= ['{:.3f}'.format(x) for x in vals] hfp.write('\n' + ','.join(strlist)) print model.summary() fp.close() hfp.close()