Perceptron from scratch
Contents
This is the most critical component of neural networks. In this section, we will learn about both forward propogation and backward propagation and the math behind back propagation.
Perceptron
This is the most fundamental type of element in a neural network. We have already seen what a perceptron is in the basics of neural networks section. However, we just scratched the surface. In this section, we will explore a perceptron in detail and explore a couple of simple problems it can solve.
Linearly Separable data
By definition, a perceptron can only solve linearly separable problems. What is a linearly separable problem ? Here are a couple of examples that show you linearly separable data. For example, two if the iris species are linearly separable by a hyperplane (in this case a single line). Similarly, an OR gate is also an example of a linearly separable dataset.
# Visualize a OR gate
import numpy as np
# OR gate data
x = np.array([[1,0],
[0,1],
[0,0],
[1,1]])
y = np.array([1,1,0,1])
import matplotlib.pyplot as plt
%matplotlib inline
plt.scatter(x[:,0],x[:,1],c=y)
plt.xlabel("x")
plt.ylabel("y")
plt.title("OR gate")
Text(0.5, 1.0, 'OR gate')
# Visualize just 2 species (setosa, versicolor) that are linearly separable
# using the predictors (Sepel Length, Sepal, Width)
import matplotlib as mpl
import matplotlib.pyplot as plt
%matplotlib inline
# iris data is readily available as a sklearn dataset.
from sklearn import datasets
iris = datasets.load_iris()
data = iris.data
# visualize just the first 100 rows (so that it contains only the species setosa and versicolor)
# We are specifically not plotting the third species (virginica), because it is not
# linearly separable.
plt.scatter(data[0:100,0],data[0:100,1],c=iris.target[0:100])
plt.xlabel("sepal length")
plt.ylabel("sepal width")
plt.title("iris species - Setosa, Versicolor")
plt.savefig("iris.png")
Now that we have an understanding of the data, let’s use gradient descent to solve for the weights.
Activation function
What we are essentially trying to do is to find out values for weights and bias in such a way that
Here is how this function would look like.
# Show how a binary step function looks like.
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
x = np.linspace (-5,5,100)
y = np.zeros(len(x))
y[x>=0] = 1
y[x<0] = 0
plt.scatter(x,y)
plt.xlabel("x")
plt.ylabel("y")
plt.title("Activation function - Binary step function ")
Text(0.5, 1.0, 'Activation function - Binary step function ')
Steps
Let’s solve the OR gate problem (or any other linearly separable problem) using a simple, single layer perceptron. Let’s start with the data first. Here is a quick high-level overview of the steps involved.
# OR gate data
x = np.array([[1,0],
[0,1],
[0,0],
[1,1]])
y = np.array([1,1,0,1])
Cost function
What about back propagation ? This is where gradient descent comes in (along with its paraphernelia of partial derivatives, learning rate, cost function etc). There are a couple of options to calculate the cost function (residual sum of squares & cross entropy), but for now, let’s just use the residual sum of squares (RSS) cost function. We have already seen this in the gradient descent section.
Partial Derivatives
Now, we want to see what the derivative of the cost function with respect to each of the variables (weights and bias).
similarly, the partial derivatives with respect to weight 2 (w2) and the bias (b) are
Update Rules
Once we have the partial derivatives, we can update the weights and biases
These equations can rattle any ML engineer, but remember, all of this is left to the library (tensorflow or any of the underlying deep learning library) to compute. The only reason why we are learning all of the math and hand-coding this in Python is to ensure that we get an in-depth understanding of back propagation. This is absolutely essential to be a good ML engineer.
Forward Propagation
Forward propagation is a relatively easy step. Let’s write a quick function for it .
def forward_prop(row) :
y_hat = np.dot(x[row],w) + b
if y_hat > 0 :
return 1
else :
return 0
Backward Propagation
Let’s now write a function for back propagation using all the geeky stuff above in “update rules” section.
def backward_prop(y_hat, row) :
global b,w
w[0] = w[0] + alpha * (y[row] - y_hat) * x[row][0]
w[1] = w[1] + alpha * (y[row] - y_hat) * x[row][1]
b = b + alpha * (y[row] - y_hat)
Initialize weights and biases
Initialize the weights and bias.
w = np.random.normal(size=2)
b = np.random.normal()
# learning rate. This is exactly the same term that we have already learnt in gradient descent.
alpha = 0.01
Predict Function
As of now, we are working on global variables to make things simple. Later, we will make a class out of all this to make things easy going forward. Just one more function to go, before we set this in motion. Assuming the model is in place (which we are going to write in a minute), we also need a function to predict a y
value, right ? Just like any Machine Learning algorithm, we need a predict ( ) method. Once the model fits the data to the right set of weights, this one is very easy. All we have to do is run the data through one forward propagation cycle.
# return the predicted y_hat, for the test data set.
def predict(x) :
y = []
# the user could be sending multiple rows. compute y_hat for each of the rows in the test dataset.
for row in x :
# weighted sum
y_pred = np.dot(row,w) + b
# run the weighted sum throught he activation function.
if y_pred > 0 :
y_pred = 1
else :
y_pred = 0
# append the predicted y (y_hat)to an array
y.append(y_pred)
# return the predicted array of y_hat values for the corresponding test data (x)
return y
Training
The individual pieces of the simple perceptron have been coded. Now, we need to write the logic to
- take the input data. For each row
- do one cycle of forward propagation
- do one cycle of backward propagation and updated the weights and bias.
This exhausts one cycle of the input data. In Deep learning, this is called as an epoch. We need to repeat the entire process for a whole bunch of epochs.
Let’s write the logic for this.
# number of epochs
for epoch in range(1000) :
# for each row in x (cycle through the dataset)
for row in range(x.shape[0]) :
# for each row in x, predict y_hat
y_hat = forward_prop(row)
# for each row calculate weights
backward_prop(y_hat,row)
print ( w, b)
[0.01363271 0.25196752] -0.009751486705392132
Predict
It is time to test our network. Let’s quickly print out x and y.
x
array([[1, 0],
[0, 1],
[0, 0],
[1, 1]])
y
array([1, 1, 0, 1])
Since this is a small dataset, we don’t need a confusion matrix to calculate the accuracy. Let’s just use the predict function on the x array to predict y.
predict(x)
[1, 1, 0, 1]
That’s a perfect match. This is a small dataset. Let’s look at a slightly larger dataset and see if the perceptron is good enough to do linear separation. Let’s pick up the iris dataset from Scikit Learn’s
from sklearn import datasets
iris = datasets.load_iris()
data = iris.data
All of this data is not linearly separable. For example, if you plot the species against the sepal length and width, the species – versicolor and virginica are muddled. Only the first species (setosa) is clearly separated.
import matplotlib.pyplot as plt
%matplotlib inline
plt.scatter(data[:,0],data[:,1],c=iris.target)
So, let’s just use the first two species. Let’s plot it again.
# visualize just the first 100 rows (so that it contains only the species setosa and versicolor)
# We are specifically not plotting the third species (virginica), because it is not
# linearly separable.
plt.scatter(data[0:100,0],data[0:100,1],c=iris.target[0:100])
plt.xlabel("sepal length")
plt.ylabel("sepal width")
plt.title("iris species - Setosa, Versicolor")
plt.savefig("iris.png")
Now, we have a clear, linear separation. Let’s train our perceptron on this data and see if it works.
x = data[0:100,0:2] # iris sepal data ( sepal length and width )
y = iris.target[0:100] # iris species data (only setosa and versicolor)
w = np.random.normal(size=2)
b = np.random.normal()
# learning rate
alpha = 0.01
# number of epochs
for epoch in range(1000) :
# for each row in x
for row in range(x.shape[0]) :
# for each row in x, predict y_hat
y_hat = forward_prop(row)
# for each row calculate weights
backward_prop(y_hat,row)
print ( w, b)
[ 0.80437979 -1.08684544] -1.0479456545593953
We can very well do a confusion matrix to check for accuracy.
y_pred = predict(x)
from sklearn.metrics import confusion_matrix
from sklearn.metrics import accuracy_score
print ( confusion_matrix(y,y_pred) )
print ( accuracy_score(y,y_pred))
[[49 1]
[ 0 50]]
0.99
That a pretty good accuracy – almost 98%. It is a bit more interesting to see this visually – using matplotlib’s meshgrid.
import numpy as np
x_all = np.linspace(0,10,100).reshape(-1,1)
y_all = np.linspace(0,10,100).reshape(-1,1)
xx,yy = np.meshgrid(x_all,y_all)
x_grid = np.concatenate((xx.ravel().reshape(-1,1),yy.ravel().reshape(-1,1)),axis=1)
x_grid
array([[ 0. , 0. ],
[ 0.1010101, 0. ],
[ 0.2020202, 0. ],
...,
[ 9.7979798, 10. ],
[ 9.8989899, 10. ],
[10. , 10. ]])
y_grid = predict(x_grid)
import matplotlib as mpl
import matplotlib.pyplot as plt
%matplotlib inline
mpl.rcParams['figure.dpi'] = 200
plt.scatter(x_grid[:,0],x_grid[:,1],c=y_grid,alpha=0.1)
plt.scatter(data[0:100,0],data[0:100,1],c=iris.target[0:100])