Deep-Learning/Chapters/10-Autoencoders/INDEX.md

# Autoencoders

Here we are trying to make a `model` to learn an **identity** function

$$
h_{\theta} (x) \approx x
$$

Now, if we were just to do this, it would be very simple, just pass
the `input` directly to `output`.

The innovation comes from the fact that we can ***compress*** `data` by using
an `NN` that has **less `neurons` per layer than `input` dimension**, or have
**less `connections` (sparse)**

## Compression

In a very simple fashion, we train a network to compress $\vec{x}$ in a more **dense**
vector $\vec{y}$ and then later **expand** it into $\vec{z}$, also called
**prediction** of $\vec{x}$

$$
\begin{aligned}
    \vec{x} &= [a, b]^{d_x} \\
    \vec{y} &= g(\vec{W_{0}}\vec{x} + b_{0}) \rightarrow \vec{y} = [a_1, b_1]^{d_y} \\
    \vec{z} &= g(\vec{W_{1}}\vec{y} + b_{1}) \rightarrow \vec{z} = [a, b]^{d_x} \\
    \vec{z} &\approx \vec{x}
\end{aligned}
$$

## Sparse Training

A sparse hidden representation comes by penalizing values assigned to `neurons`
(weights).

$$
\min_{\theta}
\underbrace{||h_{\theta}(x) - x ||^{2}}_{\text{
    Reconstruction Error
}} +
\underbrace{\lambda \sum_{i}|a_i|}_{\text{
    L1 sparsity
}}
$$

The reason on why we want **sparsity** is that we want the **best** representation
in the `latent space`, thus we want to **avoid** our `network` to **learn the
identity mapping**

## Layerwise Training

To train an `autoencoder` we train `layer` by `layer`, minimizing `vanishing gradients`.

The trick is to train one `layer`, then use it as the input for the other `layer`
and training over it as if it were our $x$. Rinse and repeat for 3 `layers` approximately.

If you want, **at last**, you can put another `layer` that you train over `data` to
**fine tune**

<!-- TODO: See Deep Belief Networks and Deep Boltzmann Machines-->
<!-- TODO: See Deep autoencoders training-->

## U-Net

It was developed to analyze medical images and segmentation, step in which we
add classification to pixels. To train these segmentation models we use **target maps**
that have the desired classification maps.

### Architecture

- **Encoder**:\
    We have several convolutional and pooling layers to make the representation smaller.
    Once small enough, we'll have a `FCNN`
- **Decoder**:\
    In this phase we restore the representation to the original dimension (`up-sampling`).
    Here we have many **deconvolution** layers, however these are learnt functions
- **Skip Connection**:\
    These are connections used to tell **deconvolutional** layers where the feature
    came from. Basically we concatenate a previous convolutional block with the
    convoluted one and we make a convolution of these layers.

<!-- TODO: See PDF anelli 10 to see complete architecture -->

## Variational Autoencoders

Until now we were reconstructing points in the latent space to points in the
**target space**.

However, these means that the **immediate neighbours of the data point** are
**meaningless**.

The idea is to make it such that all **immediate neighbour regions of our data point**
will be decoded as our **data point**.

To achieve this, our **point** will become a **distribution** over the `latent-space`
and then we'll sample from there and decode the point. We then operate as normally by
backpropagating the error.

### Regularization Term

We use `Kullback-Leibler` to see the difference in distributions. This has a
**closed form** in terms of **mean** and **covariance matrices**

The importance of regularization makes it so that these encoders are both continuous and
complete (each point is meaningfull). Without it we would have too similar results in our
regions. Also this makes it so that we don't have regions ***too concentrated and
similar to a point, nor too far apart from each other***

### Loss

$$
L(x) = ||x - \hat{x}||^{2}_{2} + KL[N(\mu_{x}, \Sigma_{x}), N(0, 1)]
$$