Added Chapter for RNNs
This commit is contained in:
Christian Risi
parent
28b6d24da6
commit
4dee495423
@ -0,0 +1,354 @@
|
|||||||
|
# Recurrent Networks | RNNs[^anelli-RNNs]
|
||||||
|
|
||||||
|
<!-- TODO: add images -->
|
||||||
|
|
||||||
|
## A bit of History[^anelli-RNNs-1]
|
||||||
|
|
||||||
|
In order to ***predict the future***, we need
|
||||||
|
***information of the past***. This is the idea behind
|
||||||
|
`RNNs` for
|
||||||
|
***predicting the next item in a `sequence`***.
|
||||||
|
|
||||||
|
While it has been attempted to accomplish this prediction
|
||||||
|
through the use of `memoryless models`, they didn't hold
|
||||||
|
up to expectations and ***had several limitations***
|
||||||
|
such as the ***dimension of the "past" window***.
|
||||||
|
|
||||||
|
- [`Autoregressive Models`](https://en.wikipedia.org/wiki/Autoregressive_model)
|
||||||
|
- [`Feed-Forward Neural Networks`](https://en.wikipedia.org/wiki/Feedforward_neural_network)
|
||||||
|
|
||||||
|
### Shortcomings of previous attempts[^anelli-RNNs-1]
|
||||||
|
|
||||||
|
- The `context window` was ***small***, thus the `model`
|
||||||
|
couldn't use ***distant past dependencies***
|
||||||
|
- Some tried to ***count words***, but it
|
||||||
|
***doesn't preserve meaning***
|
||||||
|
- Some tried to ***make the `context window` bigger***
|
||||||
|
but this
|
||||||
|
***caused words to be considered differently based
|
||||||
|
on their position***, making it
|
||||||
|
***impossible to reuse `weights` for same words***.
|
||||||
|
|
||||||
|
## RNNs[^anelli-RNNs-2]
|
||||||
|
|
||||||
|
The idea behind [`RNNs`](#rnns) is to add ***memory***
|
||||||
|
as a `hidden-state`. This helps the `model` to
|
||||||
|
***"remember"*** things for "long time", but it
|
||||||
|
is ***noisy***, and as such, the best we can do is
|
||||||
|
to ***infer its probability distribution***, doable only
|
||||||
|
for:
|
||||||
|
|
||||||
|
- [`Linear Dynamical Systems`](https://en.wikipedia.org/wiki/Linear_dynamical_system)
|
||||||
|
- [`Hidden Markov Model`](https://en.wikipedia.org/wiki/Hidden_Markov_model)
|
||||||
|
|
||||||
|
While these models are `stochastic`,
|
||||||
|
***[`RNNs`](#rnns) are `deterministic`***, plus they are ***`non-linear`*** and their
|
||||||
|
***`hidden-state` is `distributed`***[^anelli-RNNs-3]
|
||||||
|
|
||||||
|
### Neurons with Memory[^anelli-RNNs-4]
|
||||||
|
|
||||||
|
While in normal `NNs` we have no ***memory***, ***these
|
||||||
|
`neurons` have a `hidden-state`,*** $\vec{h}$ ***,
|
||||||
|
which is <u>fed back</u> to the `neuron` itself***.
|
||||||
|
|
||||||
|
<!-- TODO: Add image -->
|
||||||
|
|
||||||
|
The formula of this `hidden-state` is:
|
||||||
|
|
||||||
|
$$
|
||||||
|
\vec{h}_t = f_{W}(\vec{x}_t, \vec{h}_{t-1})
|
||||||
|
$$
|
||||||
|
|
||||||
|
In other words, ***The `hidden-state` is influenced by
|
||||||
|
a function modified by `weights`*** and
|
||||||
|
***dependent by current `inputs` and preious step
|
||||||
|
`hidden-states`***.
|
||||||
|
|
||||||
|
For example, let's say we use a $\tanh$
|
||||||
|
`activation-function`:
|
||||||
|
|
||||||
|
$$
|
||||||
|
\vec{h}_t = \tanh(
|
||||||
|
W_{h, h}^T \vec{h}_{t-1} + W_{x, h}^T \vec{x}_{t}
|
||||||
|
)
|
||||||
|
$$
|
||||||
|
|
||||||
|
And the `output` becomes:
|
||||||
|
|
||||||
|
$$
|
||||||
|
\vec{\bar{y}}_t = W_{h, y}^T \vec{h}_{t}
|
||||||
|
$$
|
||||||
|
|
||||||
|
> [!NOTE]
|
||||||
|
> Technically speaking, we could consider
|
||||||
|
> [`RNNs`](#rnns) as deep `NNs`[^anelli-RNNs-5]
|
||||||
|
|
||||||
|
#### Providing `initial-states` for the `hidden-states`[^anelli-RNNs-6]
|
||||||
|
|
||||||
|
- Specify `initial-states` of ***all*** `units`
|
||||||
|
- Specify `initial-states` for a ***subset*** of `units`
|
||||||
|
- Specify `initial-states` for the same ***subset*** of
|
||||||
|
`units` for ***each `timestep`*** (Which is the most
|
||||||
|
naural way to model sequential data)
|
||||||
|
|
||||||
|
#### Teaching signals for [`RNNs`](#rnns)[^anelli-RNNs-7]
|
||||||
|
|
||||||
|
- Specify ***desired final activity*** for ***all***
|
||||||
|
`units`
|
||||||
|
- Specify ***desired final activity*** for ***all***
|
||||||
|
`units` ofr the ***last few `steps`***
|
||||||
|
- This is good to learn `attractors`
|
||||||
|
- Makes it easy to add ***extra error derivatives***
|
||||||
|
- Speficfy the ***desired activity of a subset of
|
||||||
|
`units`***
|
||||||
|
- The other `units` will be either `inputs` or
|
||||||
|
`hidden-states`, as ***we fixed these***
|
||||||
|
|
||||||
|
#### Transforming `Data` to be used in [`RNNs`](#rnns)
|
||||||
|
|
||||||
|
- One-hot encoding: Here each `token` is a $1$ over
|
||||||
|
the `input` array
|
||||||
|
- Learned embeddings: Here each `token` is a `point`
|
||||||
|
of a ***learned hyperspace***
|
||||||
|
|
||||||
|
### Backpropagation
|
||||||
|
|
||||||
|
Since [`RNNs`](#rnns) can be considered a `deep-layered`
|
||||||
|
`NN`, then we firstly ***train the model
|
||||||
|
over the sequence*** and ***then `backpropagate`***,
|
||||||
|
keeping track of the ***training stack***, adding
|
||||||
|
derivatives along `time-steps`
|
||||||
|
|
||||||
|
> [!CAUTION]
|
||||||
|
>
|
||||||
|
> If you have ***big gradients***, remember to `clip`
|
||||||
|
> them
|
||||||
|
|
||||||
|
The thing is that is ***difficult to `train`
|
||||||
|
[`RNNs`](#rnns)*** on
|
||||||
|
***long-range dependencies*** because either the
|
||||||
|
***gradient will `vanish` or `explode`***[^anelli-RNNs-8]
|
||||||
|
|
||||||
|
> [!WARNING]
|
||||||
|
>
|
||||||
|
> `long-range dependencies` tend to have a smaller
|
||||||
|
> impact on the system than `short-range` ones
|
||||||
|
|
||||||
|
### Gated Cells
|
||||||
|
|
||||||
|
These are `neurons` that can be controlled to make
|
||||||
|
them `learn` or `forget` chosen pieces of information
|
||||||
|
|
||||||
|
> [!CAUTION]
|
||||||
|
>
|
||||||
|
> With ***chosen*** we intend choosing from the
|
||||||
|
> `hyperspace`, so it's not really precise.
|
||||||
|
|
||||||
|
#### Long Short Term Memory | LSTM[^anelli-RNNs-9][^LSTM-wikipedia]
|
||||||
|
|
||||||
|
This `cell` has a ***separate signal***, namely the
|
||||||
|
`cell-state`,
|
||||||
|
***which controls `gates` of this `cells`, always
|
||||||
|
initialized to `1`***.
|
||||||
|
|
||||||
|
> [!NOTE]
|
||||||
|
>
|
||||||
|
> $W$ will be weights associated with $\vec{x}$ and
|
||||||
|
> $U$ with $\vec{h}$.
|
||||||
|
>
|
||||||
|
> The `cell-state` has the same dimension as the
|
||||||
|
> `hidden-state`
|
||||||
|
>
|
||||||
|
> $\odot$ is the [Hadamard Product](https://en.wikipedia.org/wiki/Hadamard_product_(matrices)), also called the
|
||||||
|
> ***pointwise product***
|
||||||
|
|
||||||
|
<!-- TODO: Add images -->
|
||||||
|
|
||||||
|
##### Forget Gate | Keep Gate
|
||||||
|
|
||||||
|
This `gate` ***controls the `cell-state`***:
|
||||||
|
|
||||||
|
$$
|
||||||
|
\hat{c}_{t} = \sigma \left(
|
||||||
|
U_fh_{t-1} + W_fx_t + b_f
|
||||||
|
\right) \odot c_{t-1}
|
||||||
|
$$
|
||||||
|
|
||||||
|
The closer the result of $\sigma$ is to $0$, the more
|
||||||
|
the `cell-state` will forget that value, and opposite
|
||||||
|
for values closer to $1$.
|
||||||
|
|
||||||
|
##### Input Gate | Write Gate
|
||||||
|
|
||||||
|
***controls how much of the `input` gets into the
|
||||||
|
`cell-state`***
|
||||||
|
|
||||||
|
$$
|
||||||
|
c_{t} = \left(
|
||||||
|
\sigma \left(
|
||||||
|
U_ih_{t-1} + W_ix_t + b_i
|
||||||
|
\right) \odot \tanh \left(
|
||||||
|
U_ch_{t-1} + W_cx_t + b_c
|
||||||
|
\right)
|
||||||
|
\right) + \hat{c}_{t}
|
||||||
|
$$
|
||||||
|
|
||||||
|
The results of $\tanh$ are ***new pieces of
|
||||||
|
`information`***. The higher the $\sigma_i$, the higher
|
||||||
|
the importance given to that info.
|
||||||
|
|
||||||
|
> [!NOTE]
|
||||||
|
>
|
||||||
|
> The [`forget gate`](#forget-gate--keep-gate) and the
|
||||||
|
> [`input-gate`](#input-gate--write-gate) are 2 phases
|
||||||
|
> of the `update-phase`.
|
||||||
|
|
||||||
|
##### Output Gate | Read Gate
|
||||||
|
|
||||||
|
***Controls how much of the
|
||||||
|
`hidden-state` is forwarded***
|
||||||
|
|
||||||
|
$$
|
||||||
|
h_{t} = \tanh (c_{t}) \odot \sigma \left(
|
||||||
|
U_oh_{t-1} + W_ox_t + b_o
|
||||||
|
\right)
|
||||||
|
$$
|
||||||
|
|
||||||
|
This produces the ***new `hidden-state`***.
|
||||||
|
***Notice that
|
||||||
|
the `info` comes from the `cell-state`,
|
||||||
|
`gated` by the `input` and `previous-hidden-state`***
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
Here the `backpropagation` of the ***gradient*** is way
|
||||||
|
simpler for the `cell-states` as they ***require only
|
||||||
|
elementwise multiplications***
|
||||||
|
|
||||||
|
#### GRU[^anelli-RNNs-10][^GRU-wikipedia]
|
||||||
|
|
||||||
|
It is another type of [`gated-cell`](#gated-cells), but,
|
||||||
|
on the contrary of [`LSTM-cells`](#long-short-term-memory--lstm),
|
||||||
|
***it doesn't have a separate `cell-state`, but only
|
||||||
|
the `hidden-state`***, while keeping
|
||||||
|
***similar performances to [`LSTM`](#long-short-term-memory--lstm)***.
|
||||||
|
|
||||||
|
> [!NOTE]
|
||||||
|
> [`GRU`](#gru) doesn't have any `output-gate` and
|
||||||
|
> $h_0 = 0$
|
||||||
|
|
||||||
|
<!-- TODO: Add images -->
|
||||||
|
|
||||||
|
##### Update Gate
|
||||||
|
|
||||||
|
This `gate` unifies [`forget gate`](#forget-gate--keep-gate) and [`input gate`](#input-gate--write-gate)
|
||||||
|
|
||||||
|
$$
|
||||||
|
\begin{aligned}
|
||||||
|
|
||||||
|
\hat{h}_t &= \left(
|
||||||
|
1 - \sigma \left(
|
||||||
|
U_z h_{t-1} + W_z x_{t} + b_z
|
||||||
|
\right)
|
||||||
|
\, \right) \odot h_{t-1}
|
||||||
|
\end{aligned}
|
||||||
|
$$
|
||||||
|
|
||||||
|
##### Reset Gate
|
||||||
|
|
||||||
|
This is what breaks the `information` flow from the
|
||||||
|
previous `hidden-state`.
|
||||||
|
|
||||||
|
$$
|
||||||
|
\begin{aligned}
|
||||||
|
\bar{h}_t &= \sigma\left(
|
||||||
|
U_r h_{t-1} + W_r x_{t} + b_r
|
||||||
|
\right) \odot h_{t-1}
|
||||||
|
\end{aligned}
|
||||||
|
$$
|
||||||
|
|
||||||
|
##### New `hidden-state`
|
||||||
|
|
||||||
|
$$
|
||||||
|
\begin{aligned}
|
||||||
|
h_t = \hat{h}_t + (\sigma \left(
|
||||||
|
U_z h_{t-1} + W_z x_{t} + b_z
|
||||||
|
\right) \odot \tanh \left(
|
||||||
|
U_h \bar{h}_t + W_h x_t + b_h
|
||||||
|
\right))
|
||||||
|
\end{aligned}
|
||||||
|
$$
|
||||||
|
|
||||||
|
> [!TIP]
|
||||||
|
>
|
||||||
|
> There's no clear winner between [`GRU`](#gru) and
|
||||||
|
> [`LSTM`](#long-short-term-memory--lstm), so
|
||||||
|
> try them both, however the
|
||||||
|
> former is ***easier to compute***
|
||||||
|
|
||||||
|
### Bi-LSTM[^anelli-RNNs-12][^Bi-LSTM-stackoverflow]
|
||||||
|
|
||||||
|
It is a technique in which we put 2 `LSTM` `networks`,
|
||||||
|
***one to remember the `past` and one to remember the
|
||||||
|
`future`***.
|
||||||
|
|
||||||
|
This type of `networks` ***improve context
|
||||||
|
understanding***
|
||||||
|
|
||||||
|
### Applications[^anelli-RNNs-11]
|
||||||
|
|
||||||
|
- Music Generation
|
||||||
|
- Sentiment Classification
|
||||||
|
- Machine Translation
|
||||||
|
- Attention Mechanisms
|
||||||
|
|
||||||
|
<!-- TODO: research about Attention for RNNs -->
|
||||||
|
|
||||||
|
### Pros, Cons and Quirks
|
||||||
|
|
||||||
|
<!-- TODO: Finish this part -->
|
||||||
|
|
||||||
|
#### Pros
|
||||||
|
|
||||||
|
#### Cons
|
||||||
|
|
||||||
|
- ***hard to train***
|
||||||
|
|
||||||
|
#### Quirks
|
||||||
|
|
||||||
|
<!-- TODO: PDF 8 pg. 24 -->
|
||||||
|
|
||||||
|
<!-- Footnotes -->
|
||||||
|
|
||||||
|
[^anelli-RNNs]: Vito Walter Anelli | Deep Learning Material 2024/2025 | PDF 8
|
||||||
|
|
||||||
|
[^anelli-RNNs-1]: Vito Walter Anelli | Deep Learning Material 2024/2025 | PDF 8 pg. 11 to 20
|
||||||
|
|
||||||
|
[^anelli-RNNs-2]: Vito Walter Anelli | Deep Learning Material 2024/2025 | PDF 8 pg. 21 to 22
|
||||||
|
|
||||||
|
[^anelli-RNNs-3]: Vito Walter Anelli | Deep Learning Material 2024/2025 | PDF 8 pg. 23
|
||||||
|
|
||||||
|
<!-- TODO: find bounds of topic -->
|
||||||
|
[^anelli-RNNs-4]: Vito Walter Anelli | Deep Learning Material 2024/2025 | PDF 8 pg. 25
|
||||||
|
|
||||||
|
[^anelli-RNNs-5]: Vito Walter Anelli | Deep Learning Material 2024/2025 | PDF 8 pg. 43 to 47
|
||||||
|
|
||||||
|
[^anelli-RNNs-6]: Vito Walter Anelli | Deep Learning Material 2024/2025 | PDF 8 pg. 50
|
||||||
|
|
||||||
|
[^anelli-RNNs-7]: Vito Walter Anelli | Deep Learning Material 2024/2025 | PDF 8 pg. 51
|
||||||
|
|
||||||
|
[^anelli-RNNs-8]: Vito Walter Anelli | Deep Learning Material 2024/2025 | PDF 8 pg. 69 to 87
|
||||||
|
|
||||||
|
[^anelli-RNNs-9]: Vito Walter Anelli | Deep Learning Material 2024/2025 | PDF 8 pg. 91 to 112
|
||||||
|
|
||||||
|
[^LSTM-wikipedia]: [LSTM | Wikipedia | 27th April 2025](https://en.wikipedia.org/wiki/Long_short-term_memory)
|
||||||
|
|
||||||
|
[^anelli-RNNs-10]: Vito Walter Anelli | Deep Learning Material 2024/2025 | PDF 8 pg. 113 to 118
|
||||||
|
|
||||||
|
[^GRU-wikipedia]: [GRU | Wikipedia | 27th April 2025](https://en.wikipedia.org/wiki/Gated_recurrent_unit)
|
||||||
|
|
||||||
|
[^anelli-RNNs-11]: Vito Walter Anelli | Deep Learning Material 2024/2025 | PDF 8 pg. 119 to 126
|
||||||
|
|
||||||
|
[^anelli-RNNs-12]: Vito Walter Anelli | Deep Learning Material 2024/2025 | PDF 8 pg. 127 to 136
|
||||||
|
|
||||||
|
[^Bi-LSTM-stackoverflow]: [Bi-LSTM | StackOverflow | 27th April 2025](https://stackoverflow.com/questions/43035827/whats-the-difference-between-a-bidirectional-lstm-and-an-lstm)
|
||||||
Loading…
x
Reference in New Issue
Block a user