In parameter estimation, the performance of an estimator is usually evaluated in the respects of unbiasedness, efficiency, and consistency.
Take the estimation of parameter α for instance,
For an estimator α^, if E[α^]=α, then the estimator α^ is unbiased.
If n→∞limE[α^]=α, then the estimator α^ is progressively unbiased.
For two estimators α^1 and α^2, if Var[α^1]<Var[α^2], then α^1 is more efficient than α^2.
For an estimator α^, if ∀ϵ>0, ∃N∈Z+, when n>N, Prob[∥α^−α∥<ϵ]=1; or equivalently n→∞limProb[∥α^−α∥<ϵ]=1 holds, then the estimator α^ is consistent.
An unbiased estimator with minimal variance is termed minimal variance unbiased (MVU) estimator.
Given a random vector x, its probability density function (pdf) is denoted by p(x;θ), where θ=θ1θ2⋮ is a deterministic parameter vector. Then, we have
Likelihood function: L(θ)≜lnp(x;θ)
Score function: S(θ)≜∇θL(θ)=p(x;θ)∇θp(x;θ), where ∇θ≜∂θ1∂∂θ2∂⋮ is the gradient operator.
where ∇θθ2≜∂θ12∂2∂θ2∂θ1∂2⋮∂θ1∂θ2∂2∂θ22∂2⋮⋯⋯⋱ is the operator for Hessian matrix.
Cramer-Rao bound (CRB)
Given an arbitrary unbiased estimator of θ, denoted by θ^, E[θ^]=E[θ], its variance is lower-bounded by the inverse of FIM, a.k.a. CRB, i.e., Var[θ^]−I−1(θ)⪰0. Clearly, [Var[θ^]−I−1(θ)]ii≥0, i.e., Var[θ^i]=[Var[θ^]]ii≥[I−1(θ)]ii, i=1,2,….
If the score function can be written in the form of S(θ)=I(θ)[g(x)−θ], then θ^=g(x) is the MVU estimator, and its variance achieves minimum, i.e., Var[g(x)]=I−1(θ).