Cross Entropy

In a classification task, the end results of a DNN will be a set of a lot of 0s and one 1 which One Hot encodes the class.
To ensure that we have propabilities (ranging from 0 to 1), we apply a softmax transformation.

yiˉ=exp⁡yi^∑j=1nexp⁡yj^\bar{y_i} = \frac{\exp{\hat{y_i}}}{\sum_{j=1}^{n}\exp{\hat{y_j}}}

Now, the values represent a class probability. To avoid very small numbers, we take the negative log of the result.

−log⁡yiˉ-\log{\bar{y_i}}

In a Logistic Regression task we also use cross entropy. In this case we construct the loss with the Log-Likelihood.

ℓ(θ)=∑i=1Nlog⁡[p(yi∣xi,θ)]=⋆∑i=1Nlog⁡(μ(xi,w)yi(1−μ(xi,w))1−yi)\begin{equation} \begin{aligned} \boldsymbol{\ell}(\boldsymbol{\theta}) &=\sum_{i=1}^{N} \log \left[p\left(\mathrm{y}_{i} \mid \mathbf{x}_{i}, \boldsymbol{\theta}\right)\right] \\ & \stackrel{\star}{=} \sum_{i=1}^{N} \log \left(\mu\left(\mathbf{x}_{i}, \mathbf{w}\right)^{y_{i}}\left(1-\mu\left(\mathbf{x}_{i}, \mathbf{w}\right)\right)^{1-y_{i}}\right) \end{aligned} \end{equation} =∑i=1Nyilog⁡(μ(xi,w))+(1−yi)log⁡(1−μ(xi,w))\begin{equation}= \sum_{i=1}^{N} y_{i} \log \left(\mu\left(\mathbf{x}_{i}, \mathbf{w}\right)\right)+\left(1-y_{i}\right) \log \left(1-\mu\left(\mathbf{x}_{i}, \mathbf{w}\right)\right) \end{equation}

Now we have to minimize the negative log-likelihood and we have Cross Entropy.
This has a unique minimum but no analytical solution is possible. That is why we use Gradient Descent to get to the minimum.