How does a computer look at a photo and know there's a dog in it? Most people think it is due to the pre-programmed rules but that's not the case actually. The answer is a neural network — and understanding how it works is simpler than most people think. In this guide you will understand what a neural network is, how it learns, and why it can perform tasks that seem almost magical.
Table of Contents
- What is a Neural Network
- Input Layer
- Hidden Layer
- Output Layer
- Forward Propagation
- Activation Function
- Loss Function
- Backpropagation
- Conclusion
What is a Neural Network
A neural network is a network of interconnected artificial neurons stacked together, inspired by the human brain to perform complex tasks such as image classification, text generation, and machine translation. A simple neural network is made up of:
- Input layer
- Hidden layer
- Output layer
Each layer consists of a bunch of neurons. You can think of a neuron as a slot that holds some value.
Input Layer
It is the first layer of the neural network — the one that receives your raw data before any computation happens. Each neuron in the input layer holds exactly one number from your data. The size of the neurons in the input layer depends on the task.
Examples:
- A 28×28 grayscale image (like MNIST digits) → flattened into 784 pixel values → input layer size = 784.
- A house price model using [square footage, num bedrooms, num bathrooms, age] → input layer size = 4.
In our case we will assume the input as a 28×28 grayscale image. So the size of our input layer is 784.
Hidden Layer
The next layer after the input layer is called the hidden layer. A hidden layer can vary in size which is determined by how complex the task is. To keep things simple we assume 16 neurons in the hidden layer. To determine the values each neuron holds in the hidden layer, we have to perform a calculation on the input layer. The formula is pretty simple:
z=wx+b
The z is the sum of all the inputs multiplied with their weights plus bias. A weight is just a number that is multiplied with the input. The b is the bias that is an additional parameter alongside the weights. It provides flexibility to neural networks by allowing them to fit more complex patterns.
Initially, both weights and biases are assigned randomly. Then they are adjusted during model training.
To apply this formula in our case, let's call each value of the input layer as x₁ ,x₂ ,x₃, ....,x₇₈₄ respectively. Initially we will assign random weights w1,w2,w3,...,w784 to each neuron respectively. Let's calculate the weighted sum:
h₁ = w₁ × x₁ + w₂ × x₂ + … + w₇₈₄ × x₇₈₄ + b
We replaced z with h₁ as our hidden layer has multiple neurons. So we have to calculate it for each of the hidden neurons. In our case, the hidden layer has 16 neurons. Let's call them h1,h2,h3...,h16 respectively. The h₁ is the value of first hidden neuron. For the value of the rest of the hidden layer neurons we have to perform the same calculation with different weights and biases for each input.
To capture more complex patterns, the real neural network models use multiple hidden layers. In fact modern deep networks have ten to hundreds of hidden layers. GPT-3 for example has 96 hidden layers.
To keep things simple, let's stick to one hidden layer for now. After the hidden layer, there is the final layer called the output layer.
Output Layer
The size of the output layer is determined by the problem we are solving. Suppose we are solving a classification problem, where the neural network has to decide whether the given image is of the dog or not. So the output can only be yes or no or simple 1 or 0. For this task, there will only be one output at a time. So the output layer must have only one neuron.
To determine the value of the output neuron we have to perform the same calculation for each hidden neuron as we did for our input layer. In fact each layer in a neural network behaves like an input layer for the layer after it. So the calculation would be:
- output = h₁ × w₁ + h₂ × w₂ + h₃ × w₃ + … + h₁₆ × w₁₆ + bₕ
Forward Propagation
The process of passing input data through the layers of a neural network to generate a prediction is called the forward propagation.
Till now we have performed a forward propagation but it is not complete and perfect yet. It only captures linear patterns but real world problems are complex and non linear.
A linear network can only learn a straight-line relationship between inputs and outputs, which fails on complex real world data. In the real world there are non-linear relationships. For instance, the relationship between a person's age and their income is not a straight line — it rises, plateaus and changes at different life stages.
Activation Function
To allow our model to learn a non-linear pattern, we need to introduce non-linearity by applying the activation function to each weighted sum of every layer. Some of the most common activation functions are ReLU, Sigmoid, Tanh, Softmax.
To keep things simple let's focus on ReLU — the most widely used activation function and the easiest to understand. It also prevents the vanishing gradient problem. The formula of ReLU function is pretty simple:
f(x)=max(0,x)
For every input to the ReLU function, the negative input becomes 0, the positive inputs remain unchanged.
So our first equation will become:
h1=ReLU(w₁ × x₁ + w₂ × x₂ + … + w₇₈₄ × x₇₈₄ + b)
The same principle would be applied to the h2,h3 and so on.
Loss Function
The difference between the predicted output and the actual output is called loss. We can calculate the loss using the simple equation below:
- Loss=Prediction - Actual Output
But there is a problem with the above equation. If we try to calculate the net loss, multiple losses with negative and positive values tend to cancel each other. Let's understand it with an analogy.
Suppose our network performed two predictions 0.8 and 1.2 but their actual output was 1. According to the above equation the loss would be -0.2 and +0.2 respectively. Let's assign these losses a name l1 and l2 respectively. So the total loss would be l₁+l₂, which is 0. After calculating the total loss, it seems like the neural network made no error. That's why we use loss functions to calculate the loss such as:
- Mean Squared Error (MSE)
- Mean Absolute Error (MAE)
- Cross-Entropy Loss
Let's take MSE for now. As its name suggests, the errors are squared before performing the net loss. So the negative values also become positive. If we calculate the above losses with MSE, the net loss would be:
- Total Loss = (-0.2)² + (0.2)²= 0.08
The next step after the loss calculation is backpropagation, in which the neural network tries to reduce the loss.
BackPropagation
Backpropagation is the process of calculating gradients for each weight with respect to the loss. You might be confused with the term gradient? Once you understand it, the whole idea of how a neural network actually learns will make sense.
Let's understand with an analogy. Imagine you are adjusting one dial, and that dial affects how wrong the prediction is.
Turn the dial up a little — does the error get better or worse? Turn the dial down a little — same question. That sensitivity — how much does the error change when I nudge this dial — is the whole idea behind a gradient. It's just: which way and how hard, should I turn this knob?
A neural network does not have one knob. It has millions — every weight is a dial. The gradient for the whole network is just the answer to the same question, asked once for every single dial at a time.
In a neural network you feed in an input, the network spits out predictions, and the prediction is wrong. That's the loss that tells you how wrong.
But the loss itself does not tell you anything about which of your millions of weights caused the mistake, or which direction to move each one to fix it. That's the real problem. For the enormous amounts of weights, you have to efficiently compute how much and in which direction to change them.
Backpropagation is the algorithm that solves exactly this.
Let's say x=2, w=3, so the prediction y would be 6. Say the target was 4. So the loss would be:
loss = (6 - 4)² = 4
Now ask the knob question: If I nudge the w slightly, does the loss go up or down?
You can answer this without formal notation: y is too big, so making w smaller would decrease the loss. Making w bigger would make things worse. That's the gradient sign — which direction to move. The gradient's size tells you how sharply the loss reacts, so you know how big a step to take.
That's a gradient computed by reasoning, before any calculus.
Backpropagation efficiently calculates the gradients for each weight using the chain rule of calculus. The chain rule is a calculus technique that lets us calculate how each weight contributed to the final error, even through many layers of computation.
Once the gradient tells us which direction to move each weight, the model takes a small step in that direction. The size of that step is controlled by a value called the learning rate. Too large a step and the model overshoots. Too small and learning is painfully slow. This process of nudging every weight using its gradient is called gradient descent.
After the gradients are calculated, the model adjusts their weights and performs another forward propagation. The loss is calculated again, new gradients are computed, and the process repeats until the loss is minimized and the predictions are accurate.
Conclusion
In this blog, we did not just learned what a neural network is, but how they actually work under the hood. We also performed the actual calculation from the input layer through the output. We learned the heart of neural networks — Backpropagation.
The purpose of this whole blog was to give you a rough idea about how a simple neural network works. I kept things simple so you can understand them better. However, in real world neural network models, there are much more steps and processes involved than we discussed above such as:
- Vector Embedding
- Chunking
- Batch Processing
- Testing and Training Datasets
- OverFitting
- Matrix Multiplications
In our next articles, we will discuss each topic in detail. We will also implement a neural network in Python from scratch. Stay tuned and subscribe to our newsletter for future blogs.