How are vectors represented and used in machine learning?
A vector is an ordered collection of scalar values used to represent features, predictions, parameters, gradients, and embeddings in machine learning. Vectors support efficient computation through dot products, norms, and matrix multiplication. Their main trade-off is computational convenience versus sensitivity to inconsistent feature ordering, incompatible scales, and inappropriate dimensionality.
Read full answerHide full answer
Applies tomachine learning concepts (Python 3.12 with NumPy 2.x)
Representation and Uses
A vector of length (d) is written as (v = [v_1, v_2, ldots, v_d]), where (v_i) is the component at position (i), and (d) is the number of dimensions. In NumPy, it is commonly represented by a one-dimensional array with shape (d,).
-
Feature vector: Describes one observation, such as a customer’s age, income, and purchase history.
-
Parameter vector: Contains model weights. A linear regression with two features might use
w = [2.5, -1.2], with a separate interceptb. -
Prediction vector: Contains model outputs for one or more samples.
-
Gradient vector: Contains partial derivatives with respect to model parameters. The gradient points toward steepest increase; gradient descent moves in the opposite, negative-gradient direction.
-
Embedding vector: A learned numerical representation of an item, image, document, or other object.
Feature components must retain the same meaning and order between training and inference.
Vector Operations and Shapes
The dot product is (v cdot w = sum_{i=1}^{d} v_iw_i), where (v_i) and (w_i) are corresponding components, (d) is their common length, and the result is a scalar measuring alignment.
Two common norms are:
-
L1 norm: (|v|_1 = sum_i |v_i|), the sum of absolute component magnitudes.
-
L2 norm: (|v|_2 = sqrt{sum_i v_i^2}), the Euclidean length.
Cosine similarity is (cos(v,w) = (v cdot w)/(|v|2|w|2)). It compares direction rather than magnitude and is useful for document and embedding comparisons, although it is undefined for a zero vector.
A dataset is commonly represented by a matrix (X) with shape (n_samples, n_features). Each row is a feature vector. NumPy’s @ operator performs matrix multiplication when the inner dimensions agree, such as (n_samples, n_features) @ (n_features,). Transposing a matrix with A.T exchanges its rows and columns.
Linear Algebra Foundations
The rank of a matrix is the number of linearly independent rows or columns. Vectors are linearly independent when no vector can be expressed as a linear combination of the others. The span is the set of vectors obtainable through their linear combinations, and a basis is a minimally independent set whose span contains all vectors in that space.
For a square matrix, the determinant is a scalar associated with linear transformation. A nonzero determinant indicates an invertible matrix; a zero determinant indicates singularity. The inverse (A^{-1}) satisfies (AA^{-1}=A^{-1}A=I), where (I) is the identity matrix.
Eigenvalues, SVD, and Stability
An eigenpair satisfies (Av=lambda v), where (A) is a matrix, (v) is a nonzero eigenvector, and (lambda) is its corresponding eigenvalue. Eigenvectors describe invariant directions, while eigenvalues describe how a matrix scales or rotates along those directions. For a real symmetric matrix, np.linalg.eigh is appropriate because it returns real eigenvalues and orthogonal eigenvectors.
Singular value decomposition writes (A=USigma V^T), where (U) and (V) are orthogonal matrices, (Sigma) contains nonnegative singular values on its diagonal, and (T) denotes transpose. SVD supports dimensionality reduction, low-rank approximation, pseudo-inverses, and recommender systems.
The condition number, available through np.linalg.cond, measures sensitivity to numerical error. Ill-conditioned matrices can produce substantially different results under small input or floating-point changes.
Connections to Machine Learning
Linear regression can be expressed as (w=(X^TX)^{-1}X^Ty), where (X) contains training features, (X^T) is its transpose, (y) is the target vector, and (w) is the fitted coefficient vector. In practice, np.linalg.lstsq is preferred for least-squares problems, while np.linalg.solve is suitable for a well-conditioned square system; both avoid explicitly forming an inverse.
A neural-network layer commonly computes (Y=XW+b). Here, (X) is a batch of input vectors, (W) is a learned weight matrix, (b) is a bias vector, and (Y) contains the transformed outputs.
A symmetric matrix (A) is positive semidefinite when (x^TAx geq 0) for every vector (x). This property explains the convexity of many quadratic objectives and why covariance matrices and Hessians have special roles in optimization.
Common Pitfalls
-
Mixing features with incompatible meanings or orders invalidates learned parameters.
-
Ignoring feature scale can distort distance-based and gradient-based methods; scaling is especially important for k-NN, SVM, PCA, and many neural networks, but usually unnecessary for trees.
-
Confusing a single vector
(d,)with a batch(n_samples, d)causes shape errors. -
Treating correlated dimensions as independent can mislead interpretation and regularization.
-
Fitting scalers, encoders, or imputers on the full dataset causes data leakage; fit them only on training data, preferably inside a scikit-learn
Pipeline.