recurrent-neural-networks

Recurrent Neural Networks (RNNs)

A Recurrent Neural Network (RNN) is a class of artificial neural networks designed for processing sequential data. Unlike standard feedforward networks, RNNs possess recurrent connections, enabling them to maintain an internal state (memory) that captures dependencies across time steps.

Key Features

• Hidden State (Memory): RNNs maintain a hidden state vector $h_t$ that carries information from previous steps, updating at each time step $t$ based on the new input $x_t$ and the prior hidden state $h_{t-1}$.
• Recurrent Connections: Directed loops in the network structure allow information to persist.
• Weight Sharing: The same parameter weights are applied at each time step of the sequence, making the model computationally scalable and capable of handling variable sequence lengths.
• Backpropagation Through Time (BPTT): The network is trained by unfolding it across the sequence length and applying standard backpropagation.

Architecture & Variants

• Vanilla RNN: The basic formulation, which computes $h_t=\tanh (W_{hh}{h}_{t-1}+W_{xh}{x}_t+b_h)$. However, vanilla RNNs struggle with the vanishing and exploding gradient problems when learning long-term dependencies.
• Long Short-Term Memory (LSTM): Introduced by Hochreiter & Schmidhuber, LSTMs mitigate the vanishing gradient issue using a cell state regulated by input, forget, and output gates.
• Gated Recurrent Unit (GRU): A simplified variant of the LSTM that merges the cell state and hidden state, using update and reset gates.
• Bidirectional RNNs (BRNNs): Process input sequences in both forward and backward temporal directions to incorporate past and future context.

Applications

• Natural Language Processing (NLP): Neural machine translation, language modeling, sentiment analysis.
• Speech Recognition: Transcribing spoken language into text.
• Time Series Forecasting: Predicting financial markets, weather forecasting, and sensor anomaly detection.

Related Topics

• Deep Learning
• Transformer (which uses self-attention to process sequences in parallel, largely replacing RNNs in state-of-the-art NLP)
• Sequence-to-Sequence (Seq2Seq) models