Linear Algebra

Linear Equation

Linear Space

Linear Subspace

Affine Space

Topological Linear Space

Normed Linear Space

(1,0)-Tensor/Vector

\begin{matrix} a, b : F^{n} c : F \\ {(a + b)}_{i} \equiv a_{i} + b_{i} & , \\ {(c a)}_{i} \equiv c a_{i} & . \end{matrix}

Example

a₁ = 1, a₂ = 0, a₃ = 3 or a = (1,0, 3).

Basis

\begin{matrix} e_{i} : F^{n}, a_{i} : F & , \\ a \equiv (a_{1}, \dots, a_{n}) \equiv a_{i} e_{i} & , \\ e_{i} \cdot e_{j} \equiv case (i = j) (1) (0) \equiv {(1_{n})}_{i, j} \equiv δ_{i, j} & . \end{matrix}

Example

e_{i_{1}, i_{2}} e_{i_{2}, i_{3}} e_{i_{3}, i_{1}} = e_{i_{1}, i_{3}} e_{i_{3}, i_{1}} = e_{i_{1}, i_{1}}

Orthonormal Basis

Gram–Schmidt Process

Linear Map

\begin{matrix} A, B \subseteq f, c : f, f : A \to B \\ f (a + b) = f (a) + f (b) (additive), \\ f (c a) = c f (a) (homogeneous). \end{matrix}

Antilinear Map

f (a x + b y) = a^{*} f (x) + b^{*} f (y) .

Semilinear Map

2-Tensor/Matrix

\begin{matrix} a & , & b & : & F^{n, m} & , & c & : & f & , \\ a_{i, j} & \equiv & {(a_{i})}_{j} & , \\ ( & a & = & b & ) & \equiv & ( & i & , & j & ) & \to & ( & a_{i, j} & = & b_{i, j} & ) & . \end{matrix}

Properties

{(a + b)}_{i, j} \equiv {({(a + b)}_{i})}_{j} \equiv {(a_{i} + b_{i})}_{j} \equiv {(a_{i})}_{j} + {(b_{i})}_{j} \equiv a_{i, j} + b_{i, j} (additive).

{(c a)}_{i, j} \equiv {({(c a)}_{i})}_{j} \equiv {(c a_{i})}_{j} \equiv c {(a_{i})}_{j} \equiv c a_{i, j} (homogeneous).

a − b \equiv a + (- 1) b .

{(a - b)}_{i, j} \equiv {(a + (- 1) b)}_{i, j} \equiv a_{i, j} + {((- 1) b)}_{i, j} \equiv a_{i, j} + (- 1) b_{i, j} \equiv a_{i, j} − b_{i, j} .

Example

a₁ = (3,1) and a₂ = (0,6)or a = ((3,1), (0,6)).

Vector-Matrix Multiplication

\begin{matrix} a & : & F^{n, m} & , & x & : & F^{n} & , & y & : & F^{m} & , \end{matrix}

Left Vector Multiplication

{(x a)}_{i} \equiv x {a^{t}}_{i} .

Right Vector Multiplication

{(a y)}_{i} \equiv a_{i} y .

Matrix-Matrix Multiplication

\begin{matrix} a & : & F^{n, m} & , & b & : & F^{m, l} & , \\ a & b & : & F^{n, l} & , \\ {(a b)}_{i, j} & \equiv & a_{i, k} & b_{k, j} & \equiv & a_{i} & \cdot & {b^{t} j}_{*} & . \end{matrix}

Properties

a (b c) = (a b) c (associative) .

∵

{(a (b c))}_{i, j} \equiv a_{i, k} {(b c)}_{k, j} \equiv a_{i, k} b_{k, l} c_{l, j} \equiv {(a b)}_{i, l} c_{l, j} \equiv {((a b) c)}_{i, j} .

(a, b) × (a b ≠ b a) (non-commutative),

a (b + c) = a b + a c (left-distributive) .

∵

{(a (b + c))}_{i, j} \equiv a_{i, k} {(b + c)}_{k, j} \equiv a_{i, k} (b_{k, j} + c_{k, j}) \equiv a_{i, k} b_{k, j} + a_{i, k} c_{k, j} \equiv {(a b)}_{i, j} + {(a c)}_{i, j} \equiv {(a b + a c)}_{i, j} .

(a + b) c = a c + b c (right-distributive) .

∵

{((a + b) c)}_{i, j} \equiv {(a + b)}_{i, k} c_{k, j} \equiv (a_{i, k} + b_{i, k}) c_{k, j} \equiv a_{i, k} c_{k, j} + b_{i, k} c_{k, j} \equiv {(a c)}_{i, j} + {(b c)}_{i, j} \equiv {(a c + b c)}_{i, j} .

Matrix Exponential

Similarity

b = p^{- 1} a p

Column Matrix/Vector

\begin{matrix} a & : & F^{n, 1} & , \\ a & \equiv & ( & ( & a_{1, 1} & ) & , ⋯, & ( & a_{n, 1} & ) & ) \end{matrix}

Row Matrix/Vector

\begin{matrix} a & : & F^{1, n} & , \\ a \equiv ((a_{1, 1}, ⋯, a_{1, n}),) \end{matrix}

Transposition

{(a^{t})}_{i, j} \equiv a_{j, i} .

Identity Matrix

\begin{matrix} 1_{n} & : & F^{n, n} & , \\ {(1_{n})}_{i, j} & \equiv & case & ( & i & = & j & ) & ( & 1 & ) & ( & 0 & ) & . \end{matrix}

Zero Matrix

\begin{matrix} 0_{n} & : & F^{n, n} & , \\ {(0_{n})}_{i, j} & \equiv & 0. \end{matrix}

Scalar Matrix

\begin{matrix} c & : & f & , \\ c & 1_{n} & . \end{matrix}

Diagonal

(i ≠ j) \to (a_{i, j} = 0) .

Properties

((i ≠ j) \to (a_{i, j} = 0) × (b_{i, j} = 0)) \to ((i ≠ j) \to ({(a b)}_{i, j} = 0)) .

(i = j)→(j ≠ k)→a_i,j b_j,k = 0(j = k)→(i ≠ j)→a_i,j b_j,k = 0

Symmetric Matrix

\begin{matrix} a & : & F^{n, n} & , \\ ( & i & , & j & ) & \to & ( & a_{i, j} & = & a_{j, i} & ) & , \end{matrix}

a = a^{t} .

Properties

((i, j) \to (a_{i, j} = a_{j, i}) × (b_{i, j} = b_{j, i})) \to (i, j) \to ({(a + b)}_{i, j} = {(a + b)}_{j, i}) .

∵

{(a + b)}_{i, j} \equiv a_{i, j} + b_{i, j} = a_{j, i} + b_{j, i} \equiv {(a + b)}_{j, i}

((i, j) \to (a_{i, j} = a_{j, i})) \to (i, j) \to ({(c a)}_{i, j} = {(c a)}_{j, i}) .

((i, j) \to ({(a b)}_{i, j} = {(a b)}_{j, i})) \to (a b = b a) .

((i, j) \to (a_{i, j} = a_{j, i})) \to (i, j) \to ({a^{n}}_{i, j} = {a^{n}}_{j, i}) .

a^{- 1} = {(a^{- 1})}^{t} ≃ (a = a^{t}) × a^{- 1} exists.

Lemma

(a+a^t)^t=a^t+(a^t)^t=a^t+a=a+a^t.

Antisymmetric/skew-symmetric

\begin{matrix} a & : & F^{n, n} & , \\ ( & i & , & j & ) & \to & a_{i, j} & = & {−a}_{j, i} & , \\ or \\ a & = & a^{t} & . \end{matrix}

[a, b] \equiv a b - b a .

{[a, b]}^{t} \equiv {(a b - b a)}^{t} \equiv b^{t} a^{t} − a^{t} b^{t} = (−b) (−a) − (−a) (−b) \equiv b a - a b \equiv −[a, b].

exp (a) \equiv {(n!)}^{- 1} a^{n} .

Lemma

{(a - a^{t})}^{t} = a^{t} - {a^{t}}^{t} = a^{t} - a = - (a - a^{t}) .

Conjugate Transpose

{(a^{*})}_{i, j} \equiv {(a_{j, i})}^{*} .

Hermitian Matrix

\begin{matrix} a & : & F^{n, n} & , \\ ( & i & , & j & ) & \to & ( & a_{i, j} & = & {(a_{j, i})}^{*} & ) & , \\ or & a & = & a^{*} \end{matrix}

Orthogonal Matrix

\begin{matrix} a & : & F^{n, n} \\ a_{i, j} & a_{i, j} & = & a_{j, i} & a_{j, i} & = & {(1_{n})}_{i, j} & , \\ or & a & a^{t} & = & a^{t} & a & = & 1_{n} & , \end{matrix}

a_{j, i} = {(a^{- 1})}_{i, j} or a^{t} = a^{- 1}

Normal Matrix

\begin{matrix} a & : & F^{n, n} \\ a_{i, j} & {(a_{i, j})}^{*} & = & {(a_{j, i})}^{*} & a_{j, i} & , \\ or & a & a^{*} & = & a^{*} & a & . \end{matrix}

Unitary Matrix

\begin{matrix} a & : & F^{n, n} \\ a_{i, j} & {(a_{i, j})}^{*} & = & {(a_{j, i})}^{*} & a_{j, i} & = & {(1_{n})}_{i, j} & , \\ or & a & a^{*} & = & a^{*} & a & = & 1_{n} & . \end{matrix}

Lower/Left Triangular Matrix

\begin{matrix} a & : & F^{n, m} & , \\ ( & i & , & j & ) & \to & ( & i & < & j & ) & \to & ( & a_{i, j} & = & 0 & ) & . \end{matrix}

Upper/Right Triangular Matrix

\begin{matrix} a & : & F^{n, m} & , \\ ( & i & , & j & ) & \to & ( & i & > & j & ) & \to & ( & a_{i, j} & = & 0 & ) & . \end{matrix}

Miner

m_{i, j} \equiv \det (a_{p ≠ i, q ≠ j}) .

Cofactor

c_{i, j} \equiv {(- 1)}^{i + j} m_{i, j} .

Adjugate Matrix

{a'}_{i, j} \equiv c_{j, i} or a' \equiv c^{t} .

Inverse Matrix

{(a^{- 1})}_{i, j} \equiv {\det (a)}^{- 1} {a'}_{i, j} .

Generalised Inverse

Moore–Penrose Inverse

Trace

tr (a) \equiv a_{i, i} .

Properties

tr (a^{t}) = tr (a) .

tr (a b) = tr (b a) .

tr (c a + d b) = c tr (a) + d tr (b) .

tr (a b^{t}) = a : b .

Eigenvalues and Eigenvectors

\begin{matrix} a & : & F^{n, n} & , & x & : & F^{n} & , & x & ≠ 0, & c & : & f & , \\ a_{i, j} & x_{j} & = & c & x_{i} & , \\ or & a & x & = & c & x & . \end{matrix}

Eigenspace

Jordan Normal Form

Levi-Civita Symbol

ε_{i_{1}, i_{2}, i_{3}} \equiv caseevenodd ((i_{1}, i_{2}, i_{3})) (1) (- 1) (0) .

Alternatively

ε_{i_{1}, i_{2}, i_{3}} \equiv 2^{- 1} (i_{1} − i_{2}) (i_{2} − i_{3}) (i_{3} − i_{1}) .

See .

ε_{i_{1}, i_{2}, i_{3}} ε_{i_{4}, i_{5}, i_{1}} = e_{i_{1}, i_{4}}

ε_{i₁, i₂, i₃} ε_{i₄, i₅, i₁}=e_{i₁, i₄} e_{i₂, i₅} − e_{i₁, i₅} e_{i₂, i₄}.

Inner Product

a \cdot b \equiv a_{i} b_{i} .

Frobenius Inner Product

a⋅b=a_i,j b_i,j=vec(a)^t vec(b).

Exterior Product

\begin{matrix} e_{i_{1}} & , & e_{i_{2}} & , & e_{i_{3}} & : & F^{n} & , \\ e_{i_{1}} & \land & e_{i_{2}} & \equiv & ε_{i_{1}, i_{2}, i_{3}} & e_{i_{3}} & , \\ a & \land & b & = & a_{i} & b_{j} & ε_{i, j, k} & e_{k} & = & ε_{i, j, k} & a_{i} & b_{j} & e_{k} \end{matrix}

Lemma

a ∧ (b ∧ c)=a_i₁ e_i₁ ∧ (ε_{i₂, i₃, i₄} b_i₂ c_i₃ e_i₄)=ε_{i₂, i₃, i₄} a_i₁ b_i₂ c_i₃ (e_i₁ ∧ e_i₄)=ε_{i₂, i₃, i₄} ε_{i₁, i₄, i₅} a_i₁ b_i₂ c_i₃ e_i₅=(e e−e e) ... ...=(a⋅c) b − (a⋅b) c

a,b:R³ ⊢(a⊗b)⋅a=(a⊗b)⋅b=0 ||a ⊗ b||=||a|| ||b||

Outer Product

\begin{matrix} a & : & F^{n} & , & b & : & F^{m} & , \\ {(a \otimes b)}_{i, j} & \equiv & a_{i} & b_{j} & . \end{matrix}

\begin{matrix} i & \equiv & ( & i_{1} & , ⋯, & i_{n} & ) & , & j & \equiv & ( & j_{1} & , ⋯, & j_{m} & ) & , & a & : & F^{i} & , & b & : & F^{j} & , & a & ⊗ & b & : & F^{i, j} & , \\ {(a \otimes b)}_{i, j} & \equiv & a_{i} & b_{j} & . \end{matrix}

Example

a=(a₁, a₂, a₃), b=(b₁, b₂),a ⊗ b=((a₁ b₁, a₁ b₂), (a₂ b₁, a₂ b₂), (a₃ b₁, a₃ b₂)).

Properties

{(a \otimes b)}^{t} = b ⊗ a .

∵

^t=b ⊗ a(a⊗b)^t_i,j=(a⊗b)_j,i=a_j b_i=b_i a_j=(b ⊗ a)_i,j.

a ⊗ (b + c) = a ⊗ b + a ⊗ c (left-distributive).

∵

(a ⊗ (b+c))_i,j=a_i (b+c)_j=a_i (b_j + c_j)=a_i b_j + a_i c_j=(a⊗b)_i,j + (a ⊗ c)_i,j=(a ⊗ b+a ⊗ c)_i,j.

(a + b) ⊗ c = a ⊗ c + b ⊗ c (right-distributive).

∵

((a+b) ⊗ c)_i,j=(a+b)_i c_j=(a_i + b_i) c_j=a_i c_j + b_i c_j=(a ⊗ c)_i,j + (b⊗c)_i,j=(a ⊗ c+b ⊗ c)_i,j.

c (a \otimes b) = (c a) ⊗ b = a ⊗ (c b) .

∵

(c (a⊗b))_i,j=c (a⊗b)_i,j=c a_i b_j=(c a_i) b_j=a_i (c b_j)

Outer product of tensors satisfies:

(a⊗b) ⊗ c=a ⊗ (b⊗c) (associative).tr(a⊗b)=(a⊗b)_i,i=a_i b_ia ⊗_Kron b=vec(b ⊗_outer a)a ⊗_Kron b^t=a ⊗_outer b

Kronecker Product

a:f^n,m, b:f^l,k ⊢(a⊗b)_i,j≡a_i,j b

Example

(a₁, a₂) ⊗ (b₁, b₂)=(a₁ (b₁, b₂), a₂ (b₁, b₂))=((a₁ b₁, a₁ b₂), (a₂ b₁, a₂ b₂))

Properties

(a⊗b)_{p (r−1) + v,q (s−1) + w}=a_r,s b_v,w

Elementwise Product

Also called Hadamard product.

{(a \circ b)}_{i, j} \equiv a_{i, j} b_{i, j} .

Properties

{(a \circ b)}_{i, j} = a_{i, j} b_{i, j} = b_{i, j} a_{i, j} = {(b \circ a)}_{i, j} (commutative),

(a ∘ (b∘c))_i,j=a_i,j (b∘c)_i,j=a_i,j b_i,j c_i,j=(a∘b)_i,j c_i,j=((a∘b) ∘ c)_i,j(associative),(a ∘ (b+c))_i,j=a_i,j (b+c)_i,j=a_i,j (b_i,j + c_i,j)=a_i,j b_i,j + a_i,j c_i,j=(a ∘ b+a ∘ c)_i,j(distributive),((c a) ∘ b)_i,j=(c a)_i,j b_i,j=c a_i,j b_i,j=a_i,j (c b)_i,j=c (a_i,j b_i,j),a_i,j 0=0 a_i,j=0 or a ∘ 0_n=0_n ∘ a=0.

Khatri–Rao Product

a * b \equiv {(a_{i, j} ⊗ b_{i, j})}_{i, j} .

Example

a=((a_1,1, a_1,2), (a_2,1, a_2,2))a_1,1=((a_{1,1, 1,1}, a_{1,1, 1,2}), (a_{1,1, 2,1}, a_{1,1, 2,2}))a_1,2=((a_{1,2, 1,1},), (a_{1,2, 2,1},))a_2,1=((a_{2,1, 1,1}, a_{2,1, 1,2}),)a_2,2=((a_{2,2, 1,1},),)b=((b_1,1, b_1,2), (b_2,1, b_2,2))b_1,1=((b_{1,1, 1,1},),)b_1,2=((b_{1,2, 1,1}, b_{1,2, 1,2}),)b_2,1=((b_{2,1, 1,1},), (b_{2,1, 2,1},))b_2,2=((b_{2,2, 1,1}, b_{2,2, 1,2}), (b_{2,2, 2,1}, b_{2,2, 2,2}))a * b=((a_1,1 ⊗ b_1,1, a_1,2 ⊗ b_1,2),(a_2,1 ⊗ b_2,1, a_2,2 ⊗ b_2,2))

Tracy–Singh Product

a ∘ b=(a_i,j ∘ b)_i,j=((a_i,j ⊗ b_k,l)_k,l)_i,j

Determinant

\det (a) \equiv ε_{*} a_{1, i} a_{n, i} .

\det (a) = ε_{1_{1}, \dots, 1_{n}} a_{1, i_{1}} a_{n, i_{n}}

Properties

\det (1_{n}) = 1 .

\det (a^{t}) = \det (a) .

\det (c a) = c^{3} \det (a) .

\det (a b) = \det (a) \det (b) .

\det (u ⊗ v) = 0 .

Characteristic Polynomial

\begin{matrix} a & : & F^{n, n} & , \\ \det & ( & x & 1_{n} & - & a & ) \end{matrix}

Rank

Matrix Decomposition

Eigendecomposition/Spectral Decomposition

\begin{matrix} a & , & q & : & F^{n, n} & , \\ ( & i & ) & \to & a_{j, k} & x_{i, k} & = & c_{i, i} & x_{i, j} & , \\ or & a & = & q & c & q^{- 1} & . \end{matrix}

a,q:f^n,n, c_i,i is eigenvalue, c_i,j=0 if i ≠ j,a_j,k x_i,k=c_i,i x_i,j for each i. or a=q c q⁻¹,

Singular Value Decomposition

Dual Space

\begin{matrix} A & \subseteq & f & , \\ A^{*} & \equiv & A & \to & f & . \end{matrix}

\begin{matrix} f & , & g & : & A^{*} & , \\ ( & f & + & g & ) & ( & a & ) & = & f & ( & a & ) & + & g & ( & a & ) & , \\ ( & c & f & ) & ( & a & ) & = & c & f & ( & a & ) & . \end{matrix}

Transpose of a linear map

Linear Form

Integration

\begin{matrix} I & ( & f & + & g & ) & = & I & ( & f & ) & + & I & ( & g & ) & , \\ I & ( & a & f & ) & = & a & I & ( & f & ) & . \end{matrix}

Bilinear Form

\begin{matrix} A & , & B & , & C & \subseteq & f & , & f & : & A & × & B & \to & C & , \\ f & ( & a & + & b & , & c & ) & = & f & ( & a & , & c & ) & + & f & ( & b & , & c & ) & , \\ f & ( & c & a & , & b & ) & = & c & f & ( & a & , & b & ) & , \\ f & ( & a & , & b & + & c & ) & = & f & ( & a & , & b & ) & + & f & ( & a & , & c & ) & , \\ f & ( & a & , & c & b & ) & = & c & f & ( & a & , & b & ) & . \end{matrix}

Antisymmetric Bilinear Form

f (a, b) = −f (a, b) .

Bilinear Map

\begin{matrix} A & \subseteq & f & , & f & : & A & × & A & \to & f \\ f & ( & a & + & b & , & c & ) & = & f & ( & a & , & c & ) & + & f & ( & b & , & c & ) & , & f & ( & c & a & , & b & ) & = & c & f & ( & a & , & b & ) & , \\ f & ( & a & , & b & + & c & ) & = & f & ( & a & , & b & ) & + & f & ( & a & , & c & ) & , & f & ( & a & , & c & b & ) & = & c & f & ( & a & , & b & ) & . \end{matrix}

Alternating Bilinear Map

Submatrix/block matrix

a_{i,j, k,l}=(a_i,j)_k,l

Change of Bases

Covariance and Contravariance

Minimal Polynomial

\begin{matrix} a & : & F^{n, n} & , \\ μ_{a} & ( & a & ) & = & 0. \end{matrix}

\begin{matrix} μ_{a} & ( & a & ) & = & 0 \\ ≃ & \det & ( & λ & 1_{n} & − & a & ) & = & 0 \\ ≃ & a & x & = & λ & x & . \end{matrix}

Cayley–Hamilton Theorem

Householder Transformation

Numerical Linear Algebra

QR Decomposition

Jacobian Matrix

Hessian Matrix

Covariance and Contravariance

Active and passive transformation

Covariant transformation

Gramian matrix

Vectorisation

vec:f_m,n→f_{m n}vec(a)_i=a_{i % m,i / m}

i	⋯	m−1	m	⋯	m+m − 1	⋯	(n−1) m	⋯	(n−1) m+m − 1
i % m	⋯	m−1	0	⋯	m−1	⋯	0	⋯	0
i / m	⋯	0	1	⋯	1	⋯	n−1	⋯	n−1

Principle Invariants

\begin{matrix} a & : & F^{n, m} & , \\ I_{a} & \equiv & tr & ( & a & ) & \equiv & a_{i, i} & , \\ {II}_{a} & \equiv & ( & {tr (a)}^{2} & − & tr & ( & a^{2} & ) & ) & / 2, \\ {III}_{a} & \equiv & \det & ( & a & ) & , \end{matrix}

D_a I_a=1_n,D_a II_a=I_a 1_n − a,D_a III_a=III_a a^−t.

Linear Algebra

Contents

Linear Equation

Linear Space

Linear Subspace

Affine Space

Topological Linear Space

Normed Linear Space

(1,0)-Tensor/Vector

Example

Basis

Example

Orthonormal Basis

Gram–Schmidt Process

Linear Map

Antilinear Map

Semilinear Map

2-Tensor/Matrix

Properties

Example

Vector-Matrix Multiplication

Left Vector Multiplication

Right Vector Multiplication

Matrix-Matrix Multiplication

Properties

Matrix Exponential

Similarity

Column Matrix/Vector

Row Matrix/Vector

Transposition

Identity Matrix

Zero Matrix

Scalar Matrix

Diagonal

Properties

Symmetric Matrix

Properties

Lemma

Antisymmetric/skew-symmetric

Lemma

Conjugate Transpose

Hermitian Matrix

Orthogonal Matrix

Normal Matrix

Unitary Matrix

Lower/Left Triangular Matrix

Upper/Right Triangular Matrix

Miner

Cofactor

Adjugate Matrix

Inverse Matrix

Generalised Inverse

Moore–Penrose Inverse

Trace

Properties

Eigenvalues and Eigenvectors

Eigenspace

Jordan Normal Form

Levi-Civita Symbol

Inner Product

Frobenius Inner Product

Exterior Product

Lemma

Outer Product

Example

Properties

Kronecker Product

Example

Properties

Elementwise Product

Properties

Khatri–Rao Product

Example

Tracy–Singh Product

Determinant

Properties

Characteristic Polynomial

Rank

Matrix Decomposition

Eigendecomposition/Spectral Decomposition