From Logistic Regression to Neural Networks

In the preceding lessons, we mastered logistic regression, the foundational discriminative model that transforms a weighted linear input ($\mathbf{w}^T \phi(\mathbf{x})$) into a probability using the sigmoid function. Logistic regression, at its core, represents a single computational unit.

This lesson explores the evolution from this single unit to a complex, multi-layered architecture: the neural network.

We will first define the structure and components of a single artificial neuron, highlighting how it directly mimics a logistic regression unit. We will then assemble these units into a full network architecture, introducing concepts like hidden layers, the forward pass, and the necessity of non-linear activation functions to ensure the network can model complex, non-linear patterns. Finally, we will delve into the Backpropagation algorithm, the sophisticated optimization technique that uses the chain rule to train all parameters across all layers simultaneously.

Neural network

In a neural network, we combine distinct neurons, each with its own set of parameters $\bold{w}$ and an activation function. The figure below exemplifies a typical neural network, comprising an input layer as the first layer, consisting of input labels $x_i$. Unlike other layers, the input layer doesn’t contain neurons and simply copies the input to the subsequent layers. The last layer represents the output layer, with output labels $\hat{y}_j$. All layers situated between the input and output layers are referred to as hidden layers. The hidden layers use labels of the form $a_k^l$, where $k$ denotes the neuron index in layer $l$. Both the hidden layers and the output layer are computational, meaning they’re comprised of neurons serving as computational units.

A neural network is composed of multiple units, with each unit resembling a logistic regression unit at its core. The crucial difference lies in the flexibility afforded by multiple units and hidden layers:

1. Logistic regression (LR) is a single-unit model that uses a sigmoid activation function to perform simple binary classification (i.e., finding a single linear decision boundary).

2. A neural network (NN) stacks these units, allowing the hidden layers to learn complex, non-linear features and representations. The key distinction lies in the flexibility of these units, which can employ various nonlinear functions on top of the weighted sum $\mathbf{x}^T \mathbf{w}$. This composition enables the NN to model highly complex relationships and classification boundaries, going far beyond the limitations of the sigmoid function in a single logistic regression unit.

The image below illustrates the concept of a neuron as the functional equivalent of a simple logistic regression model:

The figure above shows that a single neuron operates identically to a logistic regression model. This model uses the sigmoid activation function ($\sigma$) to transform the weighted input into the output $\hat{y}_1$. The image provides two equivalent ways to express this calculation:

1. Scalar notation: $\hat{y}_1 = \sigma(w_1 x_1 + w_2 x_2)$, which explicitly shows the component-wise multiplication and summation for the two features $x_1$ and $x_2$.

2. Vector notation: $\hat{y}_1 = \sigma(\mathbf{x}^T \mathbf{w})$, where $\mathbf{x}$ is the input vector and $\mathbf{w}$ is the weight vector.

The vector notation, $\mathbf{x}^T \mathbf{w}$, is preferred in machine learning for its compactness and scalability, allowing the network to handle an arbitrarily large number of features efficiently using matrix operations. Since a single neuron performs this fundamental calculation, the overall functioning of a neural network is defined by how these individual units are interconnected and how their computations are executed sequentially across the layers.

Forward pass

In a neural network, each neuron takes as input a vector consisting of the outputs from all neurons in the previous layer. Consequently, every neuron produces a real number output. For a layer $l$ containing $n_l$ neurons, the output vector of this layer comprises $n_l$ components, which act as the input for all neurons in the subsequent layer $l+1$. This arrangement ensures that each neuron in layer $l+1$ possesses $n_l$ parameters, maintaining uniformity across all neurons in the same layer. Although each neuron has its own set of parameters, they all share the same number of parameters.

Let $\bold{w}_k^{l}$ represent the parameter vector of the $k^{th}$ neuron in layer $l$, then the parameter matrix $\bold{W}^l$ can be defined as $\begin{bmatrix}\bold{w}_1^{l} &\bold{w}_2^{l}&\dots&\bold{w}_{n_l}^{l}\end{bmatrix}^T$. If all neurons in layer $l$ employ the same activation function $g^l$, and $\bold{a}^{l-1}$ denotes the output vector of layer $l-1$, the relationship between them can be expressed as:

$$\bold{a}^l=g^l(\bold{W}^l\bold{a}^{l-1})$$ ...