Language Identification

Given a text, determine the Natural Language it is written in.

Method

Build a Trigram Model

P(ci∣ci−2:i−1,ℓ)\mathrm{P}\left(\mathrm{c}_i \mid \mathrm{c}_{i-2: i-1}, \ell\right)

for every language ℓ\ell.

Apply Bayes Rule and the Markov Property to get the most likely language

ℓ∗=argmax⁡ℓ(P(ℓ∣c1:N))=argmax⁡ℓ(P(ℓ)⋅P(c1:N∣ℓ))=argmax⁡ℓ(P(ℓ)⋅∏i=1NP(ci∣ci−2:i−1,ℓ))\begin{aligned} \ell^* & =\underset{\ell}{\operatorname{argmax}}\left(P\left(\ell \mid \mathbf{c}_{1: N}\right)\right) \\ & =\underset{\ell}{\operatorname{argmax}}\left(P(\ell) \cdot P\left(\mathbf{c}_{1: N} \mid \ell\right)\right) \\ & =\underset{\ell}{\operatorname{argmax}}\left(P(\ell) \cdot \prod_{i=1}^N P\left(\mathbf{c}_{\boldsymbol{i}} \mid \mathbf{c}_{\boldsymbol{i}-2: i-1}, \ell\right)\right) \end{aligned}

The Prior Probability of the language is not a critical factor.

Intuition
When ll is not the correct language all probabilities of character sequences ci−2:ic_{i-2:i} will be low. For the correct language they will be a lot higher.