The difference
Maximum Likelihood Estimation(MLE)
\hat{\theta}_{\text{MLE}} = \arg \max_{\theta} P(X | \theta)Maximum A Posteriori Estimation(MAP)
\hat{\theta}_{\text{MAP}} = \arg \max_{\theta} \left( \frac{P(X|\theta )P(\theta)}{P(X)} \right)= \arg \max_{\theta} [P(X|\theta )P(\theta)] It is using bayes rule, however, since you are only considering the mode of the distribution, you can ignore the denominator P(X). The difference between MLE and MAP is that there is a P(θ) in the loss function, and we can regard P(θ) as regularizer.
Bayes Estimation
As following Bayes rule,
P(\theta | X) = \frac{P(X|\theta ) P(\theta)}{P(X)} = \frac{P(X|\theta ) P(\theta)}{\int P(X| \theta' ) P(\theta') d\theta'}What’s difference between Bayes Estimation and the others is that θ is defined as stochastic distribution, not constant value or vector. Therefore this fact, there is actually hyperparameters that describe the stochastic distribution that is not denoted on above formulation.
And let’s focus on denominator P(X) that is described as integral function. We cannot calculate the this integral easily. So, in order to solve the problem, some methods are suggested.
- Sampling Method
- To calculate the integral, sampling methods such as Markov Chain Monte Carlo (so called MCMC). The famous algorithms are for example, the Metropolis Hastings or Gibbs sampling.
- Analytical Solution
- By considering specific conjugate priors, it becomes possible to derive the posterior distribution analytically without the need for numerical calculation.
- Approximating the Posterior Distribution
- By approximating the posterior distribution with a simple distribution, the problem is solved. For example, methods such as variational inference and expectation propagation are used. In variational inference, the posterior distribution is approximated with a simple distribution. Additionally, expectation propagation approximates the posterior distribution by the product of simple factors and iteratively updates it.
