Neural Networks in Python

Before You Begin

We've been using the Keras API with the TensorFlow backend for our simulations. Our code often also uses the numPy and SciKit libraries because I often steal code examples that happen to have been coded using those libraries. When you get started, you will want to make sure that all the requisite libraries are installed. Pip (the Python package installer) has a function called freeze that will list all your installed libraries. This list can be dumped to a text file and used to share the list of Python packages you'll need to run any of the code we've developed:

pip freeze > requirements.txt

Now any lab member can use requirements.txt to ensure they have the requisite Python packages installed:

pip install -r requirements.txt

I've got the current list of dependencies on ubfs/Scripts/Python/requirements.txt

A Simple Feedforward Classifier Network

Keras has defined network templates, but the only one that I've seen well-documented is the 'forward model. Fortunately, the forward model is well-suited to the classification task: it takes a set of input values, feeds the values forward over a set of layers, and checks the output values in a set of binary classification units. If the output decision is strictly binary (A vs B), then one output unit will suffice (since it's a zero-sum decision). If 3+ output categories are possible, then there needs to be an output unit for each possible category.

import numpy as np
import keras
import time
import scipy.io as sio
from keras.layers import Input,Dense,Dropout
from keras.models import Model, Sequential, load_model
from keras.regularizers import l1
from keras.optimizers import Adadelta
from sklearn.model_selection import StratifiedKFold as SKF
from sklearn.model_selection import ShuffleSplit as SS

import read_activations as reader

seed=int(time.time())
np.random.seed(seed)

trainfile='av_hi.csv'
traindata=np.genfromtxt(trainfile, delimiter=',')
testdata=(traindata)

x_train=traindata[:,0:-1]
y_cat_code=testdata[:,-1]
#this next line converts the category codes of 1,2,3 into 3 separate output units so that
#a code of 1 maps to {1,0,0}, 2 maps to {0,1,0} and 3 maps to {0,0,1}
y_cat_train=keras.utils.to_categorical(y_cat_code, num_classes=3)

encoding_dim1=96
encoding_dim2=32
input_trials, inodes = x_train.shape

foldnumber=0
fp=open('perflog.txt', 'w')
hfp=open('cat_traininghist.txt', 'w')

kfold = SKF(n_splits=8, shuffle=True, random_state=seed)
for train, test in kfold.split(x_train, y_cat_code):
 foldnumber=foldnumber+1
 ada=keras.optimizers.Adadelta(lr=1, rho=0.9, decay=1e-6)

 #input layer: a series of 1000 floats from 0.0 to 1.0
 main_input=Input(shape=(inodes,), dtype='float32', name='input')

 #first dropout layer, initially, lets randomly drop 20% of values
 dropout_1=Dropout(0.2, input_shape=(inodes,), name='dense_1_drop')(main_input)

 #noise layer to mitigate overfitting
 noise_1=keras.layers.noise.GaussianNoise(0.05, input_shape=(inodes,), name='dense_1_noise')(dropout_1)

 #first hidden layer
 dense_1=Dense(encoding_dim1, activation='relu', input_dim=inodes,
       name='dense_1', kernel_regularizer=l1(0.00001))(noise_1)

 #second hidden layer
 dense_2=Dense(encoding_dim2, activation='relu', name='dense_2')(dense_1)
 #category output layer
 cat_output=Dense(3, activation='softmax', name='cat_output')(dense_2)

 #assemble the model
 model=Model(inputs=[main_input], outputs=[cat_output])
 #compile the model
 model.compile(optimizer='adadelta',
      loss=['categorical_crossentropy'],
      metrics=['accuracy'], loss_weights=[1])

 X_train=x_train[train]
 X_test=x_train[test]
 Y_cat_train=y_cat_train[train]
 Y_cat_test=y_cat_train[test]

 xrows, xcols =  X_test.shape
 history=model.fit(X_train, [Y_cat_train],
  validation_split=0.2,
  epochs=64,
  batch_size=32,
  shuffle=True,
  verbose=0)
 trainscore = model.evaluate(X_test, [Y_cat_test],  verbose=0)
 outstr='%.2f\n' % trainscore[1]
 print outstr
 fp.write(outstr)
 fname='model_' + format(foldnumber, '02') + '.h5'
 model.save(fname) #save the trained model

 #training history
 vals=history.history['val_acc']
 strlist= ['{:.3f}'.format(x) for x in vals]
 hfp.write('\n' + ','.join(strlist))

print model.summary()
fp.close()
hfp.close()