BitCodeMatrix | AI-ML Fundamentals

Vectors, Matrices, Tensors, and Matrix Operations

A hands-on guide to the data structures of machine learning: shapes, axes, indexing, reshaping, broadcasting, matrix multiplication, and tensor operations, with tested NumPy code.

What You Will Learn

What scalars, vectors, matrices, and tensors are, and how to read their shapes
How real ML data (tables, images, text, video) maps to tensor shapes
Indexing, slicing, reshaping, transposing, and adding or removing axes
Element-wise operations and the rules of broadcasting
Matrix multiplication: the shape rule, properties, and batched products
Reductions along axes, combining tensors, einsum, performance tips, and common bugs

Introduction

Every machine learning model, from a simple linear regression to a large language model, receives numbers, transforms them, and outputs numbers. Those numbers are always organised in the same few structures: scalars, vectors, matrices, and tensors. The operations that transform them, such as addition, matrix multiplication, reshaping, and reductions, are the building blocks of every model.

Understanding these structures is the most practical mathematical skill in machine learning. Most errors in model code are shape errors, and most performance gains come from replacing loops with tensor operations. Reading a research paper, a PyTorch model, or a TensorFlow tutorial becomes far easier once you can track how shapes change from one operation to the next.

This article is a hands-on companion to our earlier article, "Linear Algebra for Machine Learning", which covered the theory, including eigenvalues, PCA, and the SVD. Here the focus is practical: how data is stored, how to manipulate it, and how matrix operations work in code. Every example uses NumPy and has been run to confirm its output.

Scalars, Vectors, Matrices, and Tensors

These four structures differ in one way: the number of axes, also called dimensions or the order (sometimes rank) of the tensor. In NumPy this is the ndim attribute.

Structure	Axes (ndim)	Example Shape	Notation and Example
Scalar	0	()	Lowercase italic, e.g. a learning rate α = 0.01
Vector	1	(3,)	Lowercase bold, e.g. x = [2, 5, 1], one sample's features
Matrix	2	(2, 3)	Uppercase, e.g. X, a dataset of 2 samples and 3 features
Tensor	3 or more	(3, 224, 224)	Often uppercase bold, e.g. a colour image

Figure 1: Shapes and axes of a vector, a matrix, and a 3-D tensor

Three properties describe any tensor:

Shape: the size of each axis, written as a tuple. A matrix with 2 rows and 3 columns has shape (2, 3).
Size: the total number of elements, which is the product of the shape. A (2, 3) matrix has 6 elements.
Data type (dtype): the numeric type of the elements, such as float32, float64, or int64. All elements of a tensor share one dtype.

Axes are numbered from zero. In a matrix, axis 0 runs down the rows and axis 1 runs across the columns. Strictly speaking, scalars, vectors, and matrices are all tensors of order 0, 1, and 2, but in everyday ML language "tensor" usually means three or more axes.

Python: creating tensors and inspecting their properties

import numpy as np

s = np.array(3.5) # scalar (0-D)

v = np.array([2.0, 5.0, 1.0]) # vector (1-D)

M = np.array([[1, 2, 3],

[4, 5, 6]]) # matrix (2-D)

T = np.zeros((2, 3, 4)) # 3-D tensor

for name, x in [("scalar", s), ("vector", v), ("matrix", M), ("tensor", T)]:

print(f"{name:7} ndim={x.ndim} shape={x.shape} size={x.size} dtype={x.dtype}")

Output: the scalar has shape (), the vector (3,), the matrix (2, 3) with dtype int64, and the tensor (2, 3, 4) with 24 elements.

How Real ML Data Maps to Tensor Shapes

Knowing the conventional shape of each data type is essential, because models expect their inputs in a specific layout. By convention the first axis is almost always the batch axis (N), the number of samples processed together.

Data Type	Typical Shape	Meaning of Each Axis
Tabular data	(N, F)	Samples × features, e.g. (1000, 12)
Time series	(N, T, F)	Samples × time steps × features per step
Greyscale images	(N, H, W) or (N, 1, H, W)	Samples × height × width (× one channel)
Colour images (PyTorch)	(N, C, H, W)	Channels first: e.g. (32, 3, 224, 224)
Colour images (TensorFlow/Keras)	(N, H, W, C)	Channels last: e.g. (32, 224, 224, 3)
Tokenised text	(N, T)	Sequences × token IDs
Token embeddings	(N, T, D)	Sequences × tokens × embedding size, e.g. (8, 512, 768)
Video	(N, T, C, H, W)	Clips × frames × channels × height × width

Why It Matters in ML

A neural network layer is defined by the shapes it accepts and returns. Writing the expected shape next to each line of model code, for example "# (N, T, D)", is one of the simplest habits that prevents bugs.

Common Types of Vectors and Matrices

Type	Definition	Where It Appears in ML
Row / column vector	Shape (1, n) or (n, 1)	Explicit 2-D vectors for matrix products
Unit vector	A vector with length (L2 norm) 1	Normalised embeddings for cosine similarity
One-hot vector	All zeros except a single 1	Encoding categories and class labels
Square matrix	Same number of rows and columns	Covariance and attention score matrices
Identity matrix (I)	Ones on the diagonal, zeros elsewhere	The "do nothing" transformation; residual connections
Diagonal matrix	Non-zero values only on the diagonal	Scaling each feature independently
Symmetric matrix	Equal to its transpose: A = AT	Covariance, distance, and kernel matrices
Triangular matrix	Zeros above or below the diagonal	Causal masks in transformer decoders
Orthogonal matrix	QTQ = I; preserves lengths and angles	Rotations, weight initialisation, PCA axes
Sparse matrix	Mostly zeros, stored efficiently	Text features (bag of words), user-item ratings

Indexing and Slicing

Indexing selects individual elements; slicing selects ranges. NumPy uses zero-based indexing and the syntax start:stop, where stop is excluded. A colon on its own means "everything along this axis".

Python: indexing, slicing, and boolean masks

import numpy as np

X = np.arange(12).reshape(3, 4) # 3 samples, 4 features

print(X)

print("Row 0 :", X[0]) # first sample

print("Column 2 :", X[:, 2]) # third feature

print("Element (1, 3) :", X[1, 3])

print("Sub-matrix :\n", X[0:2, 1:3])

print("Rows where feature 0 > 3:\n", X[X[:, 0] > 3]) # boolean mask

Here X[0] returns [0, 1, 2, 3], X[:, 2] returns the column [2, 6, 10], X[1, 3] returns 7, and the boolean mask keeps the last two rows, whose first feature is greater than 3. Boolean masks are widely used to filter samples, for example to select all rows with a particular label.

Views versus copies: basic slicing returns a view that shares memory with the original array, so modifying the slice also modifies the original. Boolean and integer-array indexing return copies. Call .copy() when you need an independent array.

Reshaping and Rearranging Tensors

Reshaping changes how elements are grouped into axes without changing the elements themselves. The total number of elements must stay the same: 24 elements can become (4, 6), (2, 3, 4), or (3, 8), but never (5, 5).

Operation	NumPy	Typical Use
Reshape	x.reshape(2, 3, 4)	Regroup elements into new axes
Infer one size	x.reshape(-1, 8)	Let NumPy calculate one axis (-1)
Flatten	x.flatten() or x.reshape(-1)	Turn an image into a vector before a dense layer
Transpose / permute axes	x.transpose(0, 3, 1, 2)	Convert channels-last to channels-first
Add an axis	x[np.newaxis, :] or np.expand_dims	Add a batch axis to a single sample
Remove size-1 axes	x.squeeze()	Drop leftover axes of size 1

Python: reshaping, transposing, and adding or removing axes

import numpy as np

x = np.arange(24)

print(x.reshape(4, 6).shape) # (4, 6)

print(x.reshape(2, 3, 4).shape) # (2, 3, 4)

print(x.reshape(-1, 8).shape) # (3, 8): -1 is inferred

img = np.zeros((28, 28))

print(img.flatten().shape) # (784,)

batch = np.zeros((32, 224, 224, 3)) # NHWC (TensorFlow style)

print(batch.transpose(0, 3, 1, 2).shape) # NCHW (PyTorch style)

v = np.array([1, 2, 3])

print(v[np.newaxis, :].shape, v[:, np.newaxis].shape) # (1, 3) (3, 1)

print(np.zeros((1, 3, 1)).squeeze().shape) # (3,)

NumPy stores arrays in row-major (C) order by default, and reshape fills the new shape in that order: the last axis changes fastest. This is why reshape and transpose are not interchangeable. Reshaping a (2, 3) matrix to (3, 2) regroups the same sequence of numbers, while transposing it actually swaps rows and columns.

Element-wise Operations

Element-wise operations apply to each pair of matching elements independently. They include addition, subtraction, multiplication, division, powers, and functions such as exp, log, and the activation functions used in neural networks. The inputs must have the same shape, or shapes that are compatible under broadcasting.

Python: element-wise operations versus matrix multiplication

import numpy as np

A = np.array([[1, 2], [3, 4]])

B = np.array([[5, 6], [7, 8]])

print("A + B =\n", A + B)

print("A * B (element-wise) =\n", A * B)

print("A @ B (matrix product) =\n", A @ B)

print("2 * A =\n", 2 * A)

print("A ** 2 =\n", A ** 2)

print("np.exp(A) rounded =\n", np.exp(A).round(2))

Expression	Result	What Happens
A + B	[[6, 8], [10, 12]]	Matching elements are added
A * B	[[5, 12], [21, 32]]	Hadamard product: matching elements multiplied
A @ B	[[19, 22], [43, 50]]	Matrix product: rows of A dotted with columns of B
2 * A	[[2, 4], [6, 8]]	Scalar multiplication of every element
A ** 2	[[1, 4], [9, 16]]	Each element squared (not A @ A)

The difference between * and @ is one of the most important distinctions in NumPy and PyTorch code. The element-wise (Hadamard) product, usually written with a circled-dot symbol in papers, appears in LSTM and GRU gates and in attention masks; the matrix product powers every dense layer.

Broadcasting

Broadcasting lets NumPy, PyTorch, and TensorFlow combine tensors of different shapes without writing loops or copying data. Conceptually, the smaller tensor is stretched to match the larger one.

Figure 2: Broadcasting a (3, 1) column and a (4,) row into a (3, 4) result

Two shapes are compatible when, comparing them from the rightmost axis to the left, each pair of sizes is either equal or one of them is 1. Missing leading axes are treated as size 1. For example, (3, 1) and (4,) are aligned as (3, 1) and (1, 4), which broadcast to (3, 4). By contrast, (3, 2) and (3,) fail, because the trailing sizes 2 and 3 differ.

Python: feature standardisation with broadcasting

import numpy as np

X = np.array([[1.0, 200.0],

[2.0, 300.0],

[3.0, 400.0]]) # shape (3, 2)

mean = X.mean(axis=0) # shape (2,)

std = X.std(axis=0) # shape (2,)

X_scaled = (X - mean) / std # (3, 2) - (2,) broadcasts per column

print("mean:", mean, " std:", std.round(4))

print(X_scaled.round(4))

col = np.array([[10], [20], [30]]) # (3, 1)

row = np.array([1, 2, 3, 4]) # (4,)

print((col + row).shape) # (3, 4)

try:

np.ones((3, 2)) + np.ones(3)

except ValueError as e:

print("Error:", e)

The column means are [2, 300], and after standardisation each column has values of about −1.22, 0, and 1.22. The last line raises "operands could not be broadcast together with shapes (3,2) (3,)", which is exactly the incompatibility described above.

Why It Matters in ML

Broadcasting is how a bias vector of shape (out,) is added to every row of a batch of shape (N, out), how features are normalised, and how attention masks are applied to score matrices.

Matrix Multiplication

Matrix multiplication is the most important operation in machine learning. For A of shape (m, n) and B of shape (n, p), the product C = AB has shape (m, p), and each element is the dot product of a row of A with a column of B:

cij = Σk aik bkj (m × n) · (n × p) = (m × p)

Figure 3: How one element of a matrix product is computed, and the shape rule

The inner dimensions (n) must match, and they disappear from the result; the outer dimensions (m and p) form the output shape. Computing the product requires m × n × p multiply-add operations, which is why matrix size drives the cost of training and running models.

Special Cases of Matrix Multiplication

Operation	Shapes	Result
Dot (inner) product	(n,) · (n,)	A scalar: the sum of element-wise products
Matrix-vector product	(m, n) @ (n,)	A vector of shape (m,): one dot product per row
Matrix-matrix product	(m, n) @ (n, p)	A matrix of shape (m, p)
Outer product	(m,) outer (n,)	A matrix of shape (m, n) with aibj in each cell
Batched product	(B, m, n) @ (B, n, p)	B independent products, shape (B, m, p)

Properties to Remember

Not commutative: AB and BA are generally different and may not even have the same shape. For A (2 × 3) and B (3 × 2), AB is 2 × 2 while BA is 3 × 3.
Associative: (AB)C = A(BC). The result is the same, but the cost can differ greatly, so libraries choose the cheaper order.
Distributive: A(B + C) = AB + AC.
Transpose of a product: (AB)T = BTAT. Note the reversed order.
Identity: AI = IA = A for an identity matrix of the right size.

Python: matrix multiplication and batched products

import numpy as np

A = np.array([[1, 2, 3],

[4, 5, 6]]) # (2, 3)

B = np.array([[7, 8],

[9, 10],

[11, 12]]) # (3, 2)

C = A @ B # (2, 3) @ (3, 2) -> (2, 2)

print(C)

print("C[0,0] = 1*7 + 2*9 + 3*11 =", 1*7 + 2*9 + 3*11)

print("B @ A shape:", (B @ A).shape) # (3, 3): order matters

# Batched matrix multiplication: 8 independent (2x3)@(3x2) products

batch_A = np.random.default_rng(0).normal(size=(8, 2, 3))

batch_B = np.random.default_rng(1).normal(size=(8, 3, 2))

print("batched:", (batch_A @ batch_B).shape) # (8, 2, 2)

The result is C = [[58, 64], [139, 154]]. For arrays with more than two axes, the @ operator (np.matmul) treats the last two axes as matrices and broadcasts over the leading batch axes. This is how deep learning frameworks multiply many matrices in a single call.

A Neural Network Layer in Matrix Form

A fully connected (dense) layer applied to a batch of inputs is a single matrix multiplication followed by a broadcast addition:

Y = XW + b (N, in) @ (in, out) + (out,) → (N, out)

With a batch of 32 samples, 784 input features, and 128 neurons, X is (32, 784), W is (784, 128), b is (128,), and Y is (32, 128). Note that PyTorch's nn.Linear stores its weight with shape (out, in) and computes Y = XWT + b; the mathematics is identical, only the storage convention differs.

Transpose, Inverse, Determinant, Trace, and Rank

Operation	NumPy	Meaning
Transpose	A.T	Swaps rows and columns; shape (m, n) becomes (n, m)
Inverse	np.linalg.inv(A)	The matrix A−1 with AA−1 = I (square, non-singular matrices only)
Determinant	np.linalg.det(A)	How much A scales area or volume; zero means not invertible
Trace	np.trace(A)	Sum of the diagonal elements
Rank	np.linalg.matrix_rank(A)	Number of linearly independent rows or columns
Solve Ax = b	np.linalg.solve(A, b)	Finds x directly, faster and more stable than using the inverse

Python: inverse, determinant, trace, rank, and solving systems

import numpy as np

A = np.array([[4.0, 7.0],

[2.0, 6.0]])

I = np.eye(2)

A_inv = np.linalg.inv(A)

print("det(A) :", round(np.linalg.det(A), 4)) # 4*6 - 7*2 = 10

print("A_inv :\n", A_inv)

print("A @ A_inv = I ?", np.allclose(A @ A_inv, I))

print("trace(A) :", np.trace(A))

print("rank(A) :", np.linalg.matrix_rank(A))

b = np.array([1.0, 2.0])

x = np.linalg.solve(A, b) # solves A x = b

print("solve Ax=b :", x, " check:", np.allclose(A @ x, b))

S = np.array([[2, 1], [1, 3]])

print("symmetric :", np.array_equal(S, S.T))

Q = np.array([[0.0, -1.0], [1.0, 0.0]]) # 90-degree rotation

print("orthogonal :", np.allclose(Q.T @ Q, I))

The determinant is 10, the inverse is [[0.6, −0.7], [−0.2, 0.4]], the trace is 10, and solving Ax = b gives x = [−0.8, 0.6]. In practice, prefer np.linalg.solve or np.linalg.lstsq over computing an inverse explicitly.

Reductions Along Axes

A reduction combines many values into fewer values, for example by summing, averaging, or taking the maximum. The axis argument controls which axis is collapsed. The simplest way to remember it: the axis you name is the one that disappears.

Figure 4: Summing a (2, 3) matrix along axis 0 and along axis 1

Python: reductions, keepdims, and a row-wise softmax

import numpy as np

X = np.array([[1, 2, 3],

[4, 5, 6]])

print("sum all :", X.sum())

print("sum axis=0 :", X.sum(axis=0)) # down the rows -> per column

print("sum axis=1 :", X.sum(axis=1)) # across columns -> per row

print("mean axis=1 :", X.mean(axis=1))

print("argmax axis=1:", X.argmax(axis=1))

print("keepdims :", X.sum(axis=1, keepdims=True).shape)

logits = np.array([[2.0, 1.0, 0.1],

[0.5, 2.5, 0.2]])

e = np.exp(logits - logits.max(axis=1, keepdims=True))

probs = e / e.sum(axis=1, keepdims=True)

print("softmax rows sum to:", probs.sum(axis=1))

print("predicted class :", probs.argmax(axis=1))

Summing over axis 0 gives [5, 7, 9] (one value per column); summing over axis 1 gives [6, 15] (one value per row). The softmax example shows why keepdims=True matters: keeping the reduced axis with size 1, shape (2, 1), lets the result broadcast back against the original (2, 3) array. Each row of probabilities sums to 1, and argmax returns the predicted classes [0, 1].

Why It Matters in ML

Loss functions average over the batch axis, softmax normalises over the class axis, batch normalisation averages over the batch axis, and layer normalisation averages over the feature axis. Choosing the wrong axis produces code that runs but gives wrong results.

Combining and Splitting Tensors

Concatenate joins tensors along an existing axis. Two (2, 2) matrices concatenated along axis 0 give (4, 2); along axis 1 they give (2, 4).
Stack joins tensors along a new axis. Stacking two (2, 2) matrices gives (2, 2, 2). Stacking individual samples is how a batch is built.
Split divides a tensor into parts, for example into training, validation, and test sets.

Python: concatenate, stack, and split

import numpy as np

a = np.array([[1, 2], [3, 4]])

b = np.array([[5, 6], [7, 8]])

print(np.concatenate([a, b], axis=0).shape) # (4, 2): more rows

print(np.concatenate([a, b], axis=1).shape) # (2, 4): more columns

print(np.stack([a, b], axis=0).shape) # (2, 2, 2): new axis

parts = np.split(np.arange(10), [6, 8]) # 60/20/20 split

print([p.tolist() for p in parts])

The split produces [0 to 5], [6, 7], and [8, 9]. In real projects, shuffle the data before splitting, or split by time for time-dependent data.

Einstein Summation (einsum)

einsum expresses many tensor operations in one compact notation. Each input gets a letter for each of its axes; letters that appear in the inputs but not in the output are summed over. It is available in NumPy, PyTorch, TensorFlow, and JAX, and it is common in research code, especially for attention.

einsum Expression	Equivalent	Operation
"i,i->"	a @ b	Dot product
"ij,jk->ik"	A @ B	Matrix multiplication
"ij->ji"	A.T	Transpose
"i,j->ij"	np.outer(a, b)	Outer product
"ii->"	np.trace(A)	Trace
"bij,bjk->bik"	A @ B (batched)	Batched matrix multiplication
"btd,bsd->bts"	Q @ K.transpose(0, 2, 1)	Attention scores for each sequence

Python: einsum checked against standard operations

import numpy as np

rng = np.random.default_rng(0)

a, b = rng.normal(size=3), rng.normal(size=3)

A, B = rng.normal(size=(2, 3)), rng.normal(size=(3, 4))

X = rng.normal(size=(8, 5, 3)) # batch, tokens, features

print(np.allclose(np.einsum("i,i->", a, b), a @ b)) # dot product

print(np.allclose(np.einsum("ij,jk->ik", A, B), A @ B)) # matrix product

print(np.allclose(np.einsum("ij->ji", A), A.T)) # transpose

print(np.allclose(np.einsum("i,j->ij", a, b), np.outer(a, b))) # outer product

print(np.einsum("btd,bsd->bts", X, X).shape) # batched scores

All four checks print True, and the last line produces a (8, 5, 5) tensor: a 5 × 5 matrix of token-to-token scores for each of the 8 sequences.

Performance: Writing Fast Tensor Code

Vectorise Instead of Looping

Vectorised operations hand the work to optimised, compiled libraries such as BLAS and LAPACK on the CPU, or CUDA kernels on the GPU. A Python loop processes one element at a time with interpreter overhead.

Python: a loop versus a vectorised dot product

import time

import numpy as np

rng = np.random.default_rng(0)

x, y = rng.normal(size=1_000_000), rng.normal(size=1_000_000)

t0 = time.perf_counter()

total = 0.0

for i in range(len(x)):

total += x[i] * y[i]

loop_time = time.perf_counter() - t0

t0 = time.perf_counter()

total_vec = x @ y

vec_time = time.perf_counter() - t0

print(f"Python loop : {loop_time:.3f} s")

print(f"Vectorised : {vec_time:.5f} s")

print(f"Same result : {np.isclose(total, total_vec)}")

print(f"Speed-up : ~{loop_time / vec_time:,.0f}x")

Both approaches give the same answer, but the vectorised version is typically hundreds of times faster; the exact speed-up depends on the hardware.

Choose the Right Data Type

dtype	Bytes per Value	Typical Use
float64	8	NumPy default; scientific computing where precision matters
float32	4	Standard for training neural networks
float16 / bfloat16	2	Mixed-precision training and inference on modern GPUs
int8	1	Quantised models for fast, memory-efficient inference

Memory adds up quickly. A float32 batch of shape (32, 3, 224, 224) contains 4,816,896 values, which at 4 bytes each is about 19.3 MB. Storing it as float64 doubles that to about 38.5 MB. Converting NumPy's float64 data to float32 before training is a common and worthwhile step.

Other Performance Tips

Prefer views to copies: slicing, reshaping contiguous arrays, and transposing usually create views without copying data.
Batch your work: one large matrix multiplication is much faster than many small ones, especially on GPUs.
Keep data on one device: in PyTorch, moving tensors between CPU and GPU memory is slow, so move data once and keep it there.
Use sparse formats for sparse data: scipy.sparse matrices store only the non-zero values, saving memory and time for text and ratings data.

NumPy and PyTorch Equivalents

PyTorch tensors behave much like NumPy arrays, with added GPU support and automatic differentiation. The table below maps common operations.

Operation	NumPy	PyTorch
Create	np.array([[1, 2], [3, 4]])	torch.tensor([[1, 2], [3, 4]])
Shape	x.shape	x.shape or x.size()
Reshape	x.reshape(2, -1)	x.reshape(2, -1) or x.view(2, -1)
Reorder axes	x.transpose(0, 3, 1, 2)	x.permute(0, 3, 1, 2)
Add an axis	np.expand_dims(x, 0)	x.unsqueeze(0)
Remove size-1 axes	x.squeeze()	x.squeeze()
Concatenate	np.concatenate([a, b], axis=0)	torch.cat([a, b], dim=0)
Stack	np.stack([a, b])	torch.stack([a, b])
Matrix product	a @ b or np.matmul(a, b)	a @ b or torch.matmul(a, b)
Reduce	x.sum(axis=1)	x.sum(dim=1)
einsum	np.einsum("ij,jk->ik", a, b)	torch.einsum("ij,jk->ik", a, b)
Convert	t.numpy() turns a CPU tensor into an array	torch.from_numpy(x) turns an array into a tensor
Move to GPU	Not supported	x.to("cuda")

Common Mistakes and How to Avoid Them

Confusing * with @: * is element-wise; @ is matrix multiplication. Both may run without error on square matrices but give completely different results.
Silent broadcasting bugs: subtracting predictions of shape (N, 1) from targets of shape (N,) broadcasts to an (N, N) matrix instead of N differences. A loss computed this way runs but is wrong. Check shapes, or flatten both arrays first.
Wrong reduction axis: averaging over features instead of samples, or applying softmax across the batch instead of across classes.
Forgetting the batch axis: models expect (N, ...) input, so a single sample of shape (3, 224, 224) must become (1, 3, 224, 224).
Channels-first versus channels-last: feeding an (N, H, W, C) image batch to a PyTorch model that expects (N, C, H, W).
Reshape when you meant transpose: reshape regroups elements in memory order; it does not swap axes.
Modifying a view by accident: changing a slice also changes the original array. Use .copy() when you need independence.
Integer dtypes: arrays created from integers stay integers, so assigning 2.7 into one silently stores 2. Specify dtype=np.float32 when creating model inputs.

Quick Reference: Shape Rules

Operation	Input Shapes	Output Shape
Element-wise (+, -, *, /)	(m, n) and (m, n)	(m, n)
Broadcast with a row	(m, n) and (n,)	(m, n)
Broadcast column with row	(m, 1) and (n,)	(m, n)
Dot product	(n,) and (n,)	() scalar
Matrix-vector product	(m, n) and (n,)	(m,)
Matrix product	(m, n) and (n, p)	(m, p)
Batched matrix product	(B, m, n) and (B, n, p)	(B, m, p)
Transpose	(m, n)	(n, m)
Sum over axis 0	(m, n)	(n,)
Sum over axis 1, keepdims	(m, n)	(m, 1)
Concatenate on axis 0	(m, n) and (k, n)	(m + k, n)
Stack k arrays	k arrays of (m, n)	(k, m, n)

Conclusion

Summary in 30 Seconds

Scalars, vectors, matrices, and tensors differ only in their number of axes.
Shape, size, and dtype fully describe a tensor; the first axis is usually the batch.
Reshape regroups elements; transpose reorders axes.
Broadcasting combines compatible shapes without loops or copies.
Matrix multiplication needs matching inner dimensions: (m, n) @ (n, p) = (m, p).
Reductions remove the axis you name; einsum expresses complex operations compactly.

Vectors, matrices, and tensors are the containers of machine learning, and matrix operations are the machinery. With a clear understanding of shapes, axes, broadcasting, and the matrix multiplication rule, you can read model code fluently, debug errors quickly, and write efficient, vectorised implementations. These skills transfer directly from NumPy to PyTorch, TensorFlow, and JAX.

In upcoming articles on BitCodeMatrix, we will build on these foundations with calculus and gradients for optimisation, and then implement a neural network from scratch using only the operations covered here.

Frequently Asked Questions

What is the difference between a vector, a matrix, and a tensor? A vector has one axis, a matrix has two, and a tensor generalises to any number of axes. Scalars, vectors, and matrices are tensors of order 0, 1, and 2.

What does the shape of a tensor mean? The shape lists the size of each axis. A shape of (32, 3, 224, 224) means 32 images, each with 3 colour channels and 224 × 224 pixels.

What is the difference between element-wise multiplication and matrix multiplication? Element-wise multiplication multiplies matching elements and keeps the shape. Matrix multiplication computes dot products of rows and columns and requires matching inner dimensions.

What is broadcasting? Broadcasting automatically stretches tensors with compatible shapes so element-wise operations can be applied without loops or copying data. Shapes are compared from the right, and sizes must be equal or 1.

Why is matrix multiplication not commutative? AB combines the rows of A with the columns of B, while BA combines the rows of B with the columns of A. These are different calculations, and the results usually differ in value and often in shape.

Should I learn NumPy or PyTorch first? Start with NumPy to learn the concepts, then move to PyTorch or TensorFlow. The operations and shape rules are almost identical, so the skills transfer directly.