resolve merge conflict

[notebooks.git] / fmda / presentations / rnn_data_structure.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "e8155370-6650-4d18-b3e4-c39260d217b4",
   "metadata": {},
   "source": [
    "# Input Data Structure for RNNs\n",
    "\n",
    "## Background\n",
    "\n",
    "RNNs are a type of timeseries model that relates the outcome at time $t$ to the outcome at previous times. Like other machine learning models, training is typically done by calculating the gradient of the output with respect to the weights, or parameters, of the model. With recursive or other type of autoregressive models, the gradient calculation at time $t$ ends up depending on the gradient at $t-1, t-2, ...,$ and to $t=0$. This ends up being computationally expensive, but more importantly can lead to \"vanishing\" or \"exploding\" gradient problems, where many gradients are multiplied together and either blow up or shrink. See LINK_TO_RECURSIVE_GRADIENT_LATEX for more info...\n",
    "\n",
    "RRNs and other timeseries neural network architectures* get around this issue by approximating the gradient in more stable ways. In addition to many model architecture and hyperparameter options, timeseries neural networks use two main ways of restructuring the input data.\n",
    "\n",
    "* **Sequence Length:** The input data is divided into smaller collections of ordered events, known as sequences. When calculating the gradient with respect to the model weights, the gradient only looks back this number of timesteps. Also known as `timesteps`, `sequence_length`, or just \"sample\" in `tensorflow` functions and related literature. For a sequence length of 3, the unique sequences of length 3 are: `[1,2,3], [2,3,4], ..., [T-2, T-1, T]`, for a total number of sequences `T-timesteps+1`\n",
    "\n",
    "* **Batch size:** Sequences are grouped together into batches. The batch size determines how many sequences the network processes in a single step of calculating loss and updating weights. Used as `batch_size` in `tensorflow`.\n",
    "\n",
    "The total number of batches is therefore determined from the total number of observations $T$ and the batch size. In a single batch, the loss is typically calculated for each sequence and then averaged to produce a single value. Then, the gradient of the loss with respect to the parameters (weights and biases) is computed for each sequence in the batch. So each batch will have a single gradient calculation that is the sum of the gradients of each sequence in the batch.\n",
    "\n",
    "**Note:* these same data principles apply to more complex versions of timeseries neural network layers, such as LSTM and GRU."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "39fa2a9d-c74b-445d-a875-e7591c7a6666",
   "metadata": {},
   "source": [
    "## Stateless vs Stateful Networks\n",
    "\n",
    "RNNs have a hidden state that represents the recurrent layer output at a previous time. There is a weight and bias at each RNN cell that determines the relative contribution of the previous output to the current output. When updating weights in RNNs, there are two main types of training scheme:\n",
    "\n",
    "**Stateless:** the hidden state is reset to the initial state (often zero) at the start of each new sequence in a batch. So, the network treats each sequence independently, and no information is carried over in time between sequences. These models are simpler, but work better when time dependence is relatively short.\n",
    "* **Input Data Shape:** (`n_sequence`, `timesteps`, `features`), where `n_sequence` is total number of sequences (a function of total observed times `T` and the user choice of timesteps). The input data does NOT need to be structured with batch size in stateless RNNs.\n",
    "* **Tensorflow RNN Args:** for a stateless RNN, use the `input_shape` parameter, with `input_shape`=(`timesteps`, `features`). Then, `batch_size` can be declared in the fitting stage with `model.fit(X_train, batch_size = __)`. \n",
    "\n",
    "**Stateful:** the hidden states are carried over from one sequence to the next within a batch. Longer time dependencies can be learned in this way.\n",
    "* **Input Data Shape:** (`batch_size`, `timesteps`, `features`). In order for the hidden state to be passed between sequences, the input data must be formatted using the `batch_size` hyperparameter.\n",
    "* **Tensorflow RNN Args:** for a stateful RNN, use the `batch_input_shape` parameter, with `batch_input_shape`=(`batch_size`, `timesteps`, `features`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2fa3f116-148d-4251-a8a5-e908710bc3e4",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from tensorflow.keras.preprocessing import timeseries_dataset_from_array\n",
    "import sys\n",
    "sys.path.append(\"..\")\n",
    "from moisture_rnn import staircase, staircase_2\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "50e9011c-ee92-4bc2-bf6b-461b5bf9c662",
   "metadata": {},
   "source": [
    "## Examples\n",
    "\n",
    "### Data Description\n",
    "\n",
    "Consider $T=100$ observations of a variable in time $y_t$, so $t=1, ..., 100$. A feature matrix with $3$ features has dimensions $100\\times 3$, and must be restructured for use in RNNs. In the code below each feature is a sequential list 1, 2, ..., 100 (since python numbering starts at 0)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2c03bc7a-13c4-453c-97d1-d158c85e3b9f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# print_dict_summary(train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "03048248-0c81-491d-b0ce-a099bd984dae",
   "metadata": {},
   "outputs": [],
   "source": [
    "T=100 # total number of times obseved\n",
    "features = 3\n",
    "\n",
    "# data = np.column_stack(\n",
    "#     (np.arange(1, 101)+10, \n",
    "#      np.arange(1, 101)+20, \n",
    "#      np.arange(1, 101)+30))\n",
    "data = np.arange(1, 101).reshape(-1, 1)\n",
    "\n",
    "# Generate random response vector, needed by staircase func\n",
    "y = np.arange(1, 101).reshape(-1, 1)\n",
    "\n",
    "print(f\"Response Data Shape: {y.shape}\")\n",
    "print(\"First 10 rows\")\n",
    "print(y[0:10])\n",
    "\n",
    "# Print head of data\n",
    "print(f\"Feature Data Shape: {data.shape}\")\n",
    "print(\"First 10 rows\")\n",
    "data[0:10,:]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb559e3f-b7ca-40a6-8d64-98df3bcb0f51",
   "metadata": {},
   "source": [
    "The rows of the input data array represent all features at a single timepoint. The first digit represents the feature number, and the second digit represents time point. So value $13$ represents feature 1 at time 3."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "571efd5b-7a38-4a79-9f34-8c5c7a64933f",
   "metadata": {},
   "source": [
    "### Single Batch Example\n",
    "\n",
    "With a stateless RNN, the input data is structured to be of shape (`n_sequence`, `timesteps`, `features`). The `batch_size` is not needed to structure the data for a stateless RNN.\n",
    "\n",
    "When using functions that expect `batch_size` to structure the data, an option is to set `batch_size` to be some large number greater than the total number of observed times $T$, so that all the data is guarenteed to be in one batch. *NOTE:* here we trick the function by using a large batch size, but `batch_size` could still be declared at the fitting stage of the model.\n",
    "\n",
    "Suppose in we use `timesteps=5`, so we would get sequences of times `[1,2,3,4,5]` for the first sequence, then `[2,3,4,5,6]` for the next, and so on until `[96,97,98,99,100]`. \n",
    "\n",
    "Thus, there are `100-5+1=96` possible ordered sequences of length `5`. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e17e4dc-62be-4001-ae07-1e9a985f5b9e",
   "metadata": {},
   "source": [
    "We need to structure the input data to the RNN to be of shape (96, 5, 3). *Note:* since the model is stateless, and the sequences are treated independently, the actual order of the sequences doesn't matter."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af506b7e-7ffd-484f-a1d5-a748da8f2f6d",
   "metadata": {},
   "source": [
    "For a stateless RNN, the batches could consist of any collection of the sequences, since the sequences are indepenent. \n",
    "\n",
    "We want all of the data sequences to be in a single batch, but number of batches is not a direct user input for most built-in functions. To get around this, we make the batch size some number larger than the total number of observed times."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "865805e0-caf8-4931-b9d2-6bd64cda0189",
   "metadata": {},
   "source": [
    "We now recreate this using the custom `staircase` function, which produces the same results for the input data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e350c887-af78-4036-bd66-f0e9fe574d1d",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, y_train = staircase(data, y, timesteps = 5, datapoints = len(y), verbose=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7b51ca33-40ea-4ef4-a26d-b1a875e61313",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(X_train.shape)\n",
    "print(y_train.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e321e115-1126-4aa8-9cee-020c1e464ed0",
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.DataFrame((data))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "32140b50-12e8-4352-80fd-507db473c5c5",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(X_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b14965b3-75cf-43e9-b3fe-85900e473a86",
   "metadata": {},
   "source": [
    "### Stateful Multi-Batch Example"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ce8685b5-7e79-462b-b782-6a493dc63184",
   "metadata": {},
   "source": [
    "We now need the data in the format (`batch_size`, `timesteps`, `features`) for each batch. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "341283b8-f66a-4c6b-a314-a3df4d1f08bb",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, y_train = staircase_2(data, y, timesteps = 5, batch_size = 32, verbose=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8b5edf06-2340-4358-9824-4d501812795e",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f0905891-6b67-4a47-93f7-222dedcf74cb",
   "metadata": {},
   "source": [
    "## References\n",
    "\n",
    "https://d2l.ai/chapter_recurrent-neural-networks/bptt.html\n",
    "\n",
    "https://www.tensorflow.org/guide/keras/working_with_rnns#cross-batch_statefulness\n",
    "\n",
    "Tensorflow `timeseries_dataset_from_array` tutorial: https://www.tensorflow.org/api_docs/python/tf/keras/preprocessing/timeseries_dataset_from_array\n",
    "\n",
    "Wiki BPTT: https://en.wikipedia.org/wiki/Backpropagation_through_time#:~:text=Backpropagation%20through%20time%20(BPTT)%20is,independently%20derived%20by%20numerous%20researchers.\n",
    "\n",
    "https://machinelearningmastery.com/understanding-stateful-lstm-recurrent-neural-networks-python-keras/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b3898af7-b504-4308-87b2-54c217709a54",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}