Cramer-Rao Bound

In parameter estimation, the performance of an estimator is usually evaluated in the respects of unbiasedness, efficiency, and consistency.

Take the estimation of parameter α\alpha for instance,

  • For an estimator α^\hat{\alpha}, if E[α^]=α\mathrm{E}[\hat{\alpha}] = \alpha, then the estimator α^\hat{\alpha} is unbiased.

    • If lim⁡n→∞E[α^]=α\lim\limits_{n\to\infty}\mathrm{E}[\hat{\alpha}] = \alpha, then the estimator α^\hat{\alpha} is progressively unbiased.
  • For two estimators α^1\hat{\alpha}_1 and α^2\hat{\alpha}_2, if Var[α^1]<Var[α^2]\mathrm{Var}[\hat{\alpha}_1] < \mathrm{Var}[\hat{\alpha}_2], then α^1\hat{\alpha}_1 is more efficient than α^2\hat{\alpha}_2.
  • For an estimator α^\hat{\alpha}, if ∀ϵ>0\forall \epsilon > 0, ∃N∈Z+\exists N \in \mathbb{Z}^+, when n>Nn > N, Prob[∥α^−α∥<ϵ]=1\mathrm{Prob}[\|\hat{\alpha} - \alpha\| < \epsilon] = 1; or equivalently lim⁡n→∞Prob[∥α^−α∥<ϵ]=1\lim\limits_{n\to\infty}\mathrm{Prob}[\|\hat{\alpha} - \alpha\| < \epsilon] = 1 holds, then the estimator α^\hat{\alpha} is consistent.

An unbiased estimator with minimal variance is termed minimal variance unbiased (MVU) estimator.

Given a random vector x\mathbf{x}, its probability density function (pdf) is denoted by p(x;θ)p(\mathbf{x}; \mathbf{\theta}), where θ=[θ1θ2⋮]\mathbf{\theta} = \begin{bmatrix} \theta_1 \\ \theta_2 \\ \vdots \end{bmatrix} is a deterministic parameter vector. Then, we have

  • Likelihood function: L(θ)≜ln⁡p(x;θ)L(\theta) \triangleq \ln p(\mathbf{x}; \mathbf{\theta})
  • Score function: S(θ)≜∇θL(θ)=∇θp(x;θ)p(x;θ)\mathbf{S}(\mathbf{\theta}) \triangleq \nabla_{\mathbf{\theta}}L(\mathbf{\theta}) = \dfrac{\nabla_{\mathbf{\theta}}p(\mathbf{x}; \mathbf{\theta})}{p(\mathbf{x}; \mathbf{\theta})}, where ∇θ≜[∂∂θ1∂∂θ2⋮]\nabla_{\mathbf{\theta}} \triangleq \begin{bmatrix}\dfrac{\partial}{\partial \theta_1} \\ \dfrac{\partial}{\partial \theta_2} \\ \vdots \end{bmatrix} is the gradient operator.

    • Regularity condition: ∀θ\forall \mathbf{\theta}, E[S(θ)]=0\mathrm{E}[\mathbf{S}(\mathbf{\theta})] = \mathbf{0}.

      E[S(θ)]=∫S(θ)p(x;θ)dx=∫∇θp(x;θ)dx=∇θ∫p(x;θ)dx=∇θ1=0\begin{aligned}\mathrm{E}[\mathbf{S}(\mathbf{\theta})] &= \int \mathbf{S}(\mathbf{\theta}) p(\mathbf{x}; \mathbf{\theta}) d \mathbf{x}\\&= \int \nabla_{\mathbf{\theta}} p(\mathbf{x}; \mathbf{\theta}) d \mathbf{x}\\&= \nabla_{\mathbf{\theta}} \int p(\mathbf{x}; \mathbf{\theta}) d \mathbf{x}\\&=\nabla_{\mathbf{\theta}} 1\\&= \mathbf{0}\end{aligned}
  • Fisher information matrix (FIM)

    I(θ)=Var[S(θ)]=E[S(θ)SH(θ)]=−E[∇θL(θ)∇θT]=−E[∇θθ2L(θ)],\begin{aligned}\mathbf{I}(\mathbf{\theta}) &= \mathrm{Var}[\mathbf{S}(\mathbf{\theta})]\\&= \mathrm{E}\left[\mathbf{S}(\mathbf{\theta})\mathbf{S}^H(\mathbf{\theta})\right]\\&= -\mathrm{E}\left[\nabla_{\mathbf{\theta}} L(\mathbf{\theta}) \nabla_{\mathbf{\theta}}^T\right]\\&= -\mathrm{E}\left[\nabla_{\mathbf{\theta}\mathbf{\theta}}^2 L(\mathbf{\theta})\right],\end{aligned}

    where ∇θθ2≜[∂2∂θ12∂2∂θ1∂θ2⋯∂2∂θ2∂θ1∂2∂θ22⋯⋮⋮⋱]\nabla_{\mathbf{\theta}\mathbf{\theta}}^2 \triangleq \begin{bmatrix}\dfrac{\partial^2}{\partial \theta_1^2} & \dfrac{\partial^2}{\partial\theta_1\partial\theta_2} & \cdots \\ \dfrac{\partial^2}{\partial\theta_2\partial\theta_1} &\dfrac{\partial^2}{\partial \theta_2^2} & \cdots \\ \vdots & \vdots & \ddots \end{bmatrix} is the operator for Hessian matrix.

  • Cramer-Rao bound (CRB)

    • Given an arbitrary unbiased estimator of θ\mathbf{\theta}, denoted by θ^\hat{\mathbf{\theta}}, E[θ^]=E[θ]\mathrm{E}[\hat{\mathbf{\theta}}] = \mathrm{E}[\mathbf{\theta}], its variance is lower-bounded by the inverse of FIM, a.k.a. CRB, i.e., Var[θ^]−I−1(θ)⪰0\mathrm{Var}[\hat{\mathbf{\theta}}] - \mathbf{I}^{-1}(\mathbf{\theta}) \succeq \mathbf{0}. Clearly, [Var[θ^]−I−1(θ)]ii≥0\left[\mathrm{Var}[\hat{\mathbf{\theta}}] - \mathbf{I}^{-1}(\mathbf{\theta})\right]_{ii} \geq 0, i.e., Var[θ^i]=[Var[θ^]]ii≥[I−1(θ)]ii\mathrm{Var}[\hat{\mathbf{\theta}}_i] = \left[\mathrm{Var}[\hat{\mathbf{\theta}}]\right]_{ii} \geq \left[\mathbf{I}^{-1}(\mathbf{\theta})\right]_{ii}, i=1,2,…i = 1, 2, \ldots.
    • If the score function can be written in the form of S(θ)=I(θ)[g(x)−θ]\mathbf{S}(\mathbf{\theta}) = \mathbf{I}(\mathbf{\theta})[\mathbf{g}(\mathbf{x}) - \mathbf{\theta}], then θ^=g(x)\hat{\mathbf{\theta}} = \mathbf{g}(\mathbf{x}) is the MVU estimator, and its variance achieves minimum, i.e., Var[g(x)]=I−1(θ)\mathrm{Var}[\mathbf{g}(\mathbf{x})] = \mathbf{I}^{-1}(\mathbf{\theta}).