Build a Neural Network With Pytorch

PyTorch basics

In the previous lessons, we looked at some Pytorch code, but this time we will take a closer look at it.

Pytorch is an open-source Python deep learning framework that enables us to build and train neural networks. Pytorch is a library that helps you perform mathematical operations between matrices in their basic form because, as you will realize, deep learning is just simple linear algebra.

The fundamental building block of a Pytorch is the tensor. A tensor is an N-dimensional array. We can have an 1d array (or a vector) x=[1,2,3,4,5], a 2d-array y=[[1,2],[3,4]], and so on.

In Pytorch, these can be defined as:

X= torch.tensor([ 1,2,3,4,5]) 
Y= torch.tensor([1,2],[3,4]]) 

From there we can define almost all mathematical operations between tensors.

Z = torch.add(X,Y) 
Z = torch.matmul(X,Y) 
Z = 1 / (1+torch.exp(x)) 

Let’s revisit the neuron’s equation: $a =f(w_1*a_1 + w_2*a_2 + w_3*a_3 + b_o)$

The above equation can be easily transformed into tensor operations (remember everything in deep learning can be represented as tensors).

[a_4] = f([ w1, w2, w3] * [a1, a2,a3 ] +[b_o])

All we did here is this:

• Gather together all activations into a 1-d vector and all weights into another 1-d vector.
• Multiply them.
• Add the bias.
• Apply the sigmoid function in the result.

Note that individual numbers can also be seen as 0d tensors.

Let’s proceed with our first exercise. Using everything that you learned just now, try to program your first neuron from scratch. All necessary information and commands have already been mentioned, so all you have to do is reconstruct the above equation using Pytorch.

As a first exercise, try to code a simple neuron with three inputs in Pytorch. Initialize the weights as [0.5,0.5,0.5], the bias as 0.5, and return the output.

Build a neural network

Luckily, for us, Pytorch provides lots of ready functions so that we don’t have to build each neuron from scratch. For example, if we want to declare a layer of neurons, we can use the premade function as follows:

linear1 = nn.Linear(5, 20) 

The above command constructs a layer that inputs a 5-sized vector and outputs a 20-sized vector.

To develop a neural network, we can use the following function, which defines a sequential order of individual layers:

nn.Sequential( 
        nn.Linear(2, 3), 
        nn.Sigmoid(), 
        nn.Linear(3, 2), 
        nn.Sigmoid() 
	) 

Using five lines of code, we build a neural network that has two inputs, a hidden layer with three neurons and two outputs. Note that the linear layer does not contain the activation function, so we have to explicitly declare them as well.

Program your own neural network

Now that you are more familiar with building a neural network in Pytorch, let’s see what’s going on under the hood. We will come back to the 2-3-2 NN for simplicity.

The following is another way to define a NN in Pytorch:

class Model(nn.Module): 
    def __init__(self): 
        super(Model, self).__init__() 
        self.linear1 = nn.Linear(2, 3) 
        self.linear2 = nn.Linear(3, 2) 
  
    def forward(self, x): 
        h = torch.sigmoid(self.linear1(x)) 
        o = torch.sigmoid(self.linear2(h)) 
        return o 

Many people prefer it because it gives more control and explainability over the network. To run a forward propagation, we can create a random input, initialize the model, and pass the input as an argument as follows:

model= Model() 
X = torch.randn((1, 2)) 
Y = model(X) 

After placing print statements between all layers, we can inspect the state of our NN.

Our input is:

tensor([0.5000, 0.5000]) 

While the first layer’s weights are:

tensor([[-0.3580, -0.4130], 
    	[ 0.5652,  0.6722], 
    	[ 0.4894,  0.4164]] 

Lastly, the first layer’s output is:

tensor([0.5555, 0.4797, 0.5787]) 

Now, it is your turn to build a neural network. Let’s have it receive a vector with 10 numbers, have 3 linear layers with dimensions 128, 64, 2, and two RELU layers in between.