How do neural networks work and compare to biological brains?
A neural network is a layered system of interconnected artificial neurons that learns input-output relationships by adjusting numerical weights and biases. It resembles a biological brain because both propagate signals through weighted connections, but artificial networks are usually trained by backpropagation rather than by biological synapses. The main trade-off is expressive power versus greater data, computation, tuning, and interpretability requirements.
Read full answerHide full answer
Applies tomachine learning concepts (Python 3.12)
How It Works
Each neuron first computes a weighted sum and then applies a nonlinear activation:
z = sum(w_i * x_i) + b, followed by a = f(z)
-
x_i: input value -
w_i: learned weight or connection strength -
b: learned bias -
z: pre-activation value -
f: activation function -
a: neuron output
A network stacks these neurons into an input layer, one or more hidden layers, and an output layer. During the forward pass, each layer transforms its inputs into outputs. During backpropagation, the chain rule calculates how each weight contributed to the loss, allowing gradient-based optimizers to update it.
For example, if a sigmoid output is 0.881, the target is 1, and the pre-activation is z = 2, then dL/dz = 0.881 - 1 = -0.119. If z = w*x with x = 2, then dL/dw = -0.119 * 2 = -0.238.
The universal approximation theorem states that a sufficiently wide single-hidden-layer network can approximate many continuous functions on a compact input domain. It does not guarantee that training will find a useful solution or that it will generalize well.
Brain Analogy
Similarities - Both use weighted connections whose strengths change during learning. - Signals propagate through multiple processing stages. - Nonlinearity allows complex patterns to be represented. - Local computations combine into system-level behavior.
Important differences - Biological neurons communicate through spikes and use electrochemical synapses. - Brains learn continuously through neural plasticity, often without explicit labels. - Artificial networks commonly use backpropagation, large datasets, and deliberate optimization. - An artificial network’s similarity to the brain is an analogy, not a complete biological model.
Activation Functions
| Activation | Main use | Gradient behavior |
|---|---|---|
| Sigmoid | Binary output probabilities | Can produce vanishing gradients when inputs are large |
| Tanh | Hidden layers centered around zero | Also susceptible to vanishing gradients |
| ReLU | Common hidden-layer activation | Efficient, but can produce dying ReLU units |
| Leaky ReLU | Reduces dying-ReLU risk | Allows a small negative slope |
| GELU | Smooth transformer-style hidden layers | Smooth, differentiable approximation |
| Softmax | Multiclass output probabilities | Produces normalized class probabilities |
Training Practices and Architectures
-
Initialization: Xavier initialization suits tanh or sigmoid units; He initialization is common with ReLU.
-
Learning rate: Too high may cause divergence; too low makes training slow.
-
Batch size: Smaller batches add noise but can improve generalization; larger batches improve hardware utilization.
-
Normalization: Batch or layer normalization can stabilize optimization.
-
Dropout: Randomly disables units during training to reduce overfitting.
-
Early stopping: Stop when validation performance stops improving.
-
Scaling: Standardize numerical features, especially for gradient-based models.
-
Missing values: Impute them or use natively missing-aware models without leaking information from validation data.
| Output | Activation | Loss |
|---|---|---|
| Binary classification | Sigmoid | Binary cross-entropy |
| Multiclass classification | Softmax | Cross-entropy |
| Regression | Linear | MSE or MAE |
For imbalanced classification, inspect precision, recall, F1, PR-AUC, and ROC-AUC rather than accuracy alone. Common architectures include MLPs for tabular data, CNNs for spatial data, RNNs for sequences, and transformers for long-range dependencies.
Common Pitfalls
-
Combining sigmoid with
BCEWithLogitsLoss, which already applies sigmoid internally. -
Applying softmax before a loss that expects raw logits.
-
Scaling the entire dataset before creating training and validation splits.
-
Reusing the validation set repeatedly until performance becomes artificially optimistic.
-
Selecting a complex network for tiny datasets without regularization.
-
Assuming a GPU is always faster; small models can perform better on CPUs.
-
Reporting accuracy for severe class imbalance without additional metrics.
-
Neglecting random seeds, data provenance, and saved preprocessing artifacts.