TensorFlow

TensorFlow is an open source software library for numerical computation using data flow graphs. Nodes in the graph represent mathematical operations, while the graph edges represent the multidimensional data arrays (tensors) communicated between them. The flexible architecture allows you to deploy computation to one or more CPUs or GPUs in a desktop, server, or mobile device with a single API. TensorFlow was originally developed by researchers and engineers working on the Google Brain Team within Google's Machine Intelligence research organization for the purposes of conducting machine learning and deep neural networks research, but the system is general enough to be applicable in a wide variety of other domains as well.

Basic Python and Training Principles

TensorFlow was designed for use in Python 3.0, but has been ported to C# and a few other languages. If you don't know any Python 3.0, or are coming from Python 2.0, this website (https://www.learnpython.org/) has some very succinct tutorials that don't take much time to run through.

Here are some links showing a basic model using the numpy module:

https://iamtrask.github.io/2015/07/12/basic-python-network/
https://iamtrask.github.io/2015/07/27/python-network-part2/

This is a useful source for checking out Python code and also learning about basic model training principles.

Installation

You can find information for installing TensorFlow here: https://www.tensorflow.org/install/

Note, Ubuntu has more installation options than those described below. This section is mostly tailored for a Windows installation, which currently contains a few errors or omissions that you might not catch without some digging.

In short, if you want to utilize GPU multithreading (assuming you have a compatible GPU) you'll need to install NVIDIA CUDA 8.0 libraries, and cuDNN version 6.0 (incorrectly called 6.1 in the above link for Windows). CUDA 8.0 will first require the installation of Visual Studio 2015 (not specified in the above link) which can be found for free, but you need to make an account. The later versions of Visual Studio (e.g. 2017) are not currently supported. Before going much further you'll want to validate that your CUDA libraries are functioning and can detect your GPU. This involves navigating to the CUDA utilities dir, building deviceQuery_vs2015 and bandwidthTest_vs2015 with Visual Studio, and executing the resulting .exe files with the Windows command prompt. Those steps are described here:

http://developer2.download.nvidia.com/compute/cuda/8.0/secure/Prod2/docs/sidebar/CUDA_Toolkit_Release_Notes.pdf?oPCuqG4MnSXkQlnN1TMx4j3-52ejYtNrsuO-DolRD6QZWCSy1ssU1B9aGt-XKAwdWVtuvJqZUr0_6eMBbL8UkK9t0do6GsQvC_45womqGlLuISMoHzaZjW7THPa_Hsa4j2xnYBhFUOm46yqEwzRH5GP0EL_-juRJJZy2GKkSMQksigE8hF7nXA

Next you'll need Python 3 (3.5 or 3.6), of which there are multiple install options. If you already have an older version of Python and you don't want to risk collisions, you may want to install via Anaconda (or virtualenv for Ubuntu). Anaconda creates a virtual environment for Python to operate, and comes with the Spyder IDE, which is very similar to Matlab. However, you would be limited to operating within this environment. You'll want to make sure CUDA 8.0, cuDNN 6.0 (if you haven't dropped these files into the corresponding CUDA dirs), and your Python scripts are in your path (%path% for Windows, or .bashrc for Ubuntu). Installing on Ubuntu also requires the libcupti-dev library. Now you can install TensorFlow with pip (or conda or virtualenv). The command may vary for doing so depending on your set-up (e.g. py -m pip install tensorflow-gpu OR python -m pip install tensorflow-gpu). If you decided to do a 'native' pip install of Python 3, you'll also want to find an IDE. You could just use Notepad, but it's not 1980 and there are way better and more efficient development tools. Rodeo (developed by yhat) is a good choice, it operates very much like R studio and Matlab (so it should be somewhat familiar if coming from those). Another option is the Atom text editor (with the hydrogen package), its got some interesting tricks, but seems less useful for large data sets.

If you don't care about taking advantage of GPU acceleration, or have a dedicated machine for doing so, you can just skip ahead and install Python 3 (or 2.7, a final update of 2.0 to make it more compatible with 3.0, but isn't compatible with CUDA).

Once you've got everything set-up, the following should output "Hello, TensorFlow!"

import tensorflow as tf
hello = tf.constant('Hello, TensorFlow!')
sess = tf.Session()
print(sess.run(hello))

If you get an error indicating that TensorFlow can't be found, check your paths. Or, if you did a 'native' pip install, make sure your IDE is operating with the Python kernel.

Using TensorFlow (with Keras)

In Progress...

Unlike most other programming languages, Python has very few core functions and instead relies on modules selected by the user to perform particular functions. As such, you don't actually need to know a ton about Python except for basic syntax, vectors and arrays, how modules work, and how to structure blocks. Most of the functions you'll use for TensorFlow (or Keras) come from the package itself.

Keras is a high level API that runs on top of TensorFlow, Theano, or CNTK. It's supposed to be very user friendly and easy to use.

py -m pip install keras 

Toy XOR Network

Here's a simple and easily customized network that solves the exclusive-or problem.

from keras.models import Sequential #
from keras.layers.core import Dense, Dropout, Activation #
from keras.optimizers import SGD #
import numpy as np 

X = np.array([[0,0],[0,1],[1,0],[1,1]]) #Input
y = np.array([[0],[1],[1],[0]]) #Target Output

model = Sequential()
model.add(Dense(8, input_dim=2)) #
model.add(Activation('tanh')) # 
model.add(Dense(1)) #
model.add(Activation('sigmoid'))

sgd = SGD(lr=0.1, momentum=0.1) #Configure Optimization 
model.compile(loss='mean_squared_error', optimizer=sgd) #Set error correction method (e.g. 'mean_squared_error' or 'binary_crossentropy')

model.fit(X, y, batch_size=4, epochs=1000) #larger batches should require fewer epochs
print(model.predict_proba(X))

Smart XOR Network

from keras.models import Sequential from keras.layers.core import Dense, Dropout, Activation from keras.optimizers import SGD # from keras.callbacks import Callback # from keras.initializers import VarianceScaling # import numpy as np

1. Configure stopping point (error less than .008)

lastEpoch = 0 class EarlyStoppingByLossVal(Callback):

   def __init__(self, monitor='val_loss', value=0.008, verbose=0):
       super(Callback, self).__init__()
       self.monitor = monitor
       self.value = value
       self.verbose = verbose
   def on_epoch_end(self, epoch, logs={}):
       global lastEpoch
       current = logs.get("loss")         
       if current != None and current < self.value:
           self.model.stop_training = True
           lastEpoch = epoch + 1

1. Provide Input Layer and Output Layer

X = np.array([[0,0],[0,1],[1,0],[1,1]]) #Input y = np.array([[0],[1],[1],[0]]) #Target Output

1. Build the Model

model = Sequential() model.add(Dense(8, input_dim=2,use_bias = False, kernel_initializer = VarianceScaling())) # model.add(Activation('tanh')) # model.add(Dense(1, use_bias = False, kernel_initializer = VarianceScaling())) # model.add(Activation('sigmoid')) #Activation function

sgd = SGD(lr=0.4, momentum=0.4) #Configure Optimization model.compile(loss='mean_squared_error', optimizer=sgd) #Set error correction method (e.g. 'mean_squared_error' or 'binary_crossentropy')

model.fit(X, y, verbose=1, batch_size=4, epochs=1000, callbacks = [EarlyStoppingByLossVal()]) #larger batches should require fewer epochs print(model.predict_proba(X)) #Display output weights print("Last epoch: " + str(lastEpoch)) #Display last epoch