| blob | cb756a142c408abbd595450691f20d3bdf2e1af9 |
1 {
2 "nbformat": 4,
3 "nbformat_minor": 0,
4 "metadata": {
5 "colab": {
6 "name": "rnn_tutorial_jm.ipynb",
7 "provenance": [],
8 "collapsed_sections": []
9 },
10 "kernelspec": {
11 "name": "python3",
12 "display_name": "Python 3"
13 },
14 "language_info": {
15 "name": "python"
16 }
17 },
18 "cells": [
19 {
20 "cell_type": "markdown",
21 "metadata": {
22 "id": "hEhvqXQBNN6r"
23 },
24 "source": [
25 "# Understanding Simple Recurrent Neural Networks In Keras\n",
26 "From https://machinelearningmastery.com/understanding-simple-recurrent-neural-networks-in-keras/ \n",
27 "With changes by Jan Mandel\n",
28 "2021-12-26: small changes for clarity\n",
29 "2022-06-16: added same by functional interface"
30 ]
31 },
32 {
33 "cell_type": "markdown",
34 "metadata": {
35 "id": "pINycjukQijP"
36 },
37 "source": [
38 "This tutorial is designed for anyone looking for an understanding of how recurrent neural networks (RNN) work and how to use them via the Keras deep learning library. While all the methods required for solving problems and building applications are provided by the Keras library, it is also important to gain an insight on how everything works. In this article, the computations taking place in the RNN model are shown step by step. Next, a complete end to end system for time series prediction is developed.\n",
39 "\n",
40 "After completing this tutorial, you will know:\n",
41 "\n",
42 "* The structure of RNN\n",
43 "* How RNN computes the output when given an input\n",
44 "* How to prepare data for a SimpleRNN in Keras\n",
45 "* How to train a SimpleRNN model\n",
46 "\n",
47 "Let’s get started."
48 ]
49 },
50 {
51 "cell_type": "markdown",
52 "metadata": {
53 "id": "3D7sqhF-QtJ4"
54 },
55 "source": [
56 "## Tutorial Overview\n",
57 "\n",
58 "This tutorial is divided into two parts; they are:\n",
59 "\n",
60 "1. The structure of the RNN\n",
61 " 1. Different weights and biases associated with different layers of the RNN.\n",
62 " 2. How computations are performed to compute the output when given an input.\n",
63 "2. A complete application for time series prediction."
64 ]
65 },
66 {
67 "cell_type": "markdown",
68 "metadata": {
69 "id": "jU0opwkmQ9wm"
70 },
71 "source": [
72 "## Prerequisites\n",
73 "\n",
74 "It is assumed that you have a basic understanding of RNNs before you start implementing them. An [Introduction To Recurrent Neural Networks And The Math That Powers Them](https://machinelearningmastery.com/an-introduction-to-recurrent-neural-networks-and-the-math-that-powers-them) gives you a quick overview of RNNs.\n",
75 "\n",
76 "Let’s now get right down to the implementation part."
77 ]
78 },
79 {
80 "cell_type": "markdown",
81 "metadata": {
82 "id": "-UcYCr1FMaE7"
83 },
84 "source": [
85 "## Import section"
86 ]
87 },
88 {
89 "cell_type": "code",
90 "metadata": {
91 "id": "FimVgAB4LsI9"
92 },
93 "source": [
94 "from pandas import read_csv\n",
95 "import numpy as np\n",
96 "from keras.models import Sequential\n",
97 "from keras.layers import Dense, SimpleRNN\n",
98 "from sklearn.preprocessing import MinMaxScaler\n",
99 "from sklearn.metrics import mean_squared_error\n",
100 "import math\n",
101 "import matplotlib.pyplot as plt\n",
102 "import tensorflow as tf"
103 ],
104 "execution_count": null,
105 "outputs": []
106 },
107 {
108 "cell_type": "markdown",
109 "metadata": {
110 "id": "O-okDDv9Md4Z"
111 },
112 "source": [
113 "## Keras SimpleRNN"
114 ]
115 },
116 {
117 "cell_type": "markdown",
118 "metadata": {
119 "id": "W-WQZp0qRta0"
120 },
121 "source": [
122 "The function below returns a model that includes a SimpleRNN layer and a Dense layer for learning sequential data. The input_shape specifies the parameter (time_steps x features). We’ll simplify everything and use univariate data, i.e., one feature only; the time_steps are discussed below."
123 ]
124 },
125 {
126 "cell_type": "code",
127 "source": [
128 "api_type = 2 # 1 = sequential, 2 = functional"
129 ],
130 "metadata": {
131 "id": "Qf35beQp1Ewv"
132 },
133 "execution_count": null,
134 "outputs": []
135 },
136 {
137 "cell_type": "code",
138 "metadata": {
139 "id": "Aoh0LWQ7Ltdk"
140 },
141 "source": [
142 "def create_RNN_sequential(hidden_units, dense_units, input_shape, activation):\n",
143 " model = Sequential()\n",
144 " model.add(SimpleRNN(hidden_units, input_shape=input_shape, \n",
145 " activation=activation[0]))\n",
146 " model.add(Dense(units=dense_units, activation=activation[1]))\n",
147 " model.compile(loss='mean_squared_error', optimizer='adam')\n",
148 " return model"
149 ],
150 "execution_count": null,
151 "outputs": []
152 },
153 {
154 "cell_type": "code",
155 "source": [
156 "def create_RNN_functional(hidden_units, dense_units, input_shape=None, activation=None,\n",
157 " return_sequences=False,stateful=False,batch_shape=None):\n",
158 " if stateful:\n",
159 " inputs = tf.keras.Input(batch_shape=batch_shape)\n",
160 " else:\n",
161 " inputs = tf.keras.Input(shape=input_shape)\n",
162 " #inputs = tf.keras.Input(shape=input_shape)\n",
163 " x = tf.keras.layers.SimpleRNN(hidden_units,\n",
164 " return_sequences=return_sequences,\n",
165 " stateful=stateful,\n",
166 " activation=activation[0])(inputs)\n",
167 " outputs = tf.keras.layers.Dense(dense_units, activation=activation[1])(x)\n",
168 " model = tf.keras.Model(inputs=inputs, outputs=outputs)\n",
169 " model.compile(loss='mean_squared_error', optimizer='adam')\n",
170 " return model"
171 ],
172 "metadata": {
173 "id": "zBQxW0h8pvGb"
174 },
175 "execution_count": null,
176 "outputs": []
177 },
178 {
179 "cell_type": "code",
180 "source": [
181 "def create_RNN(hidden_units, dense_units, input_shape, activation):\n",
182 " if api_type==1:\n",
183 " print('using sequential api')\n",
184 " return create_RNN_sequential(hidden_units, dense_units, input_shape, activation)\n",
185 " if api_type==2:\n",
186 " print('using functional api')\n",
187 " return create_RNN_functional(hidden_units, dense_units, input_shape, activation)\n",
188 " print('api_type must be 1 or 2, got ',api_type)\n",
189 " raise(ValueError)\n",
190 "\n",
191 "demo_model = create_RNN(hidden_units=2, dense_units=1, input_shape=(3,1), \n",
192 " activation=['linear', 'linear'])"
193 ],
194 "metadata": {
195 "id": "dYLai9HypVRo"
196 },
197 "execution_count": null,
198 "outputs": []
199 },
200 {
201 "cell_type": "markdown",
202 "metadata": {
203 "id": "_CZtOyDPSuMy"
204 },
205 "source": [
206 "The object demo_model is returned with 2 hidden units created via a the SimpleRNN layer and 1 dense unit created via the Dense layer. The input_shape is set at 3×1 and a linear activation function is used in both layers for simplicity. Just to recall the linear activation function f(x)=x makes no change in the input."
207 ]
208 },
209 {
210 "cell_type": "code",
211 "source": [
212 "print(dir(demo_model))\n",
213 "# help(demo_model)\n",
214 "help(demo_model.get_weights)"
215 ],
216 "metadata": {
217 "id": "TgDQmxwQpUtw"
218 },
219 "execution_count": null,
220 "outputs": []
221 },
222 {
223 "cell_type": "markdown",
224 "metadata": {
225 "id": "BI0vVCjQTBPF"
226 },
227 "source": [
228 "Look at the model following https://machinelearningmastery.com/visualize-deep-learning-neural-network-model-keras/ :"
229 ]
230 },
231 {
232 "cell_type": "code",
233 "metadata": {
234 "id": "Rdz2nblaTMV5"
235 },
236 "source": [
237 "print(demo_model.summary())\n",
238 "from keras.utils.vis_utils import plot_model\n",
239 "plot_model(demo_model, to_file='model_plot.png', \n",
240 " show_shapes=True, show_layer_names=True)"
241 ],
242 "execution_count": null,
243 "outputs": []
244 },
245 {
246 "cell_type": "markdown",
247 "metadata": {
248 "id": "6KMS82uNTZz4"
249 },
250 "source": [
251 "If we have m\n",
252 " hidden units (\n",
253 "m\n",
254 "=\n",
255 "2\n",
256 " in the above case), then:\n",
257 "* Input: \n",
258 "x\n",
259 "∈\n",
260 "R\n",
261 "* Hidden unit: \n",
262 "h\n",
263 "∈\n",
264 "R<sup>m</sup>\n",
265 "* Weights for input units: \n",
266 "w<sub>x</sub>\n",
267 "∈\n",
268 " R<sup>m</sup>\n",
269 "* Weights for hidden units: \n",
270 "w<sub>h</sub>\n",
271 "∈\n",
272 "R<sup>m x m</sup>\n",
273 "* Bias for hidden units: \n",
274 "b<sub>h</sub>\n",
275 "∈\n",
276 "R\n",
277 "<sup>m</sup>\n",
278 "*Weight for the dense layer: \n",
279 "w<sub>y</sub>\n",
280 "∈\n",
281 "R\n",
282 "<sup>m</sup>\n",
283 "*Bias for the dense layer: \n",
284 "b<sub>y</sub>\n",
285 "∈\n",
286 "R\n",
287 "\n",
288 "Let’s look at the above weights. The weights are generated randomly so they will be different every time. The important thing is to learn what the structure of each object being used looks like and how it interacts with others to produce the final output. The get_weights() method of the model object returns a list of arrays, which consists of the weights and the bias of each layer, in the order of the layers. The first layer's input takes two entries, the (external) input values and the values of the hidden variables from the previous step."
289 ]
290 },
291 {
292 "cell_type": "code",
293 "metadata": {
294 "id": "f-C4rYMDL1_l"
295 },
296 "source": [
297 "w = demo_model.get_weights()\n",
298 "#print(len(w),' weight arrays:',w)\n",
299 "wname=('wx','wh','bh','wy','by','wz','bz')\n",
300 "for i in range(len(w)):\n",
301 " print(i,':',wname[i],'shape=',w[i].shape)\n",
302 "\n",
303 "wx = w[0]\n",
304 "wh = w[1]\n",
305 "bh = w[2]\n",
306 "wy = w[3]\n",
307 "by = w[4]"
308 ],
309 "execution_count": null,
310 "outputs": []
311 },
312 {
313 "cell_type": "code",
314 "source": [
315 "# help(SimpleRNN)"
316 ],
317 "metadata": {
318 "id": "kNzpNlwP4MSs"
319 },
320 "execution_count": null,
321 "outputs": []
322 },
323 {
324 "cell_type": "markdown",
325 "metadata": {
326 "id": "oks2sHZlUQZB"
327 },
328 "source": [
329 "Now let’s do a simple experiment to see how the layers from a SimpleRNN and Dense layer produce an output. Keep this figure in view.\n",
330 "<img src=\"https://machinelearningmastery.com/wp-content/uploads/2021/09/rnnCode1.png\">"
331 ]
332 },
333 {
334 "cell_type": "markdown",
335 "metadata": {
336 "id": "eGi9cUkgUkTe"
337 },
338 "source": [
339 "We’ll input x for three time steps and let the network generate an output. The values of the hidden units at time steps 1, 2 and 3 will be computed. \n",
340 "h<sub>0</sub>\n",
341 " is initialized to the zero vector. The output \n",
342 "o<sub>3</sub>\n",
343 " is computed from \n",
344 "h<sub>3</sub>\n",
345 " and \n",
346 "w<sub>3</sub>\n",
347 ". An activation function is linear, f(x)=x, so the update of $h(k)$ and the output $o(k)$ are given by"
348 ]
349 },
350 {
351 "cell_type": "markdown",
352 "metadata": {
353 "id": "QSD8ABJXxS7K"
354 },
355 "source": [
356 "\\begin{align*}\n",
357 "h\\left( 0\\right) = &\\left[\n",
358 "\\begin{array}\n",
359 "[c]{c}%\n",
360 "0\\\\\n",
361 "0\n",
362 "\\end{array}\n",
363 "\\right] \\\\\n",
364 "h\\left( k+1\\right) =&x\\left( k\\right) \\left[\n",
365 "\\begin{array}\n",
366 "[c]{c}%\n",
367 "w_{x,0}\\\\\n",
368 "w_{x,1}%\n",
369 "\\end{array}\n",
370 "\\right] +\\left[ h_{0}(k),h_{1}(k)\\right] \\left[\n",
371 "\\begin{array}\n",
372 "[c]{cc}%\n",
373 "w_{h,00} & w_{h,01}\\\\\n",
374 "w_{h,10} & w_{h,11}%\n",
375 "\\end{array}\n",
376 "\\right] +\\left[\n",
377 "\\begin{array}\n",
378 "[c]{c}%\n",
379 "b_{h,0}\\\\\n",
380 "b_{h,1}%\n",
381 "\\end{array}\n",
382 "\\right] \\\\\n",
383 "o(k+1)=& \\left[ h_{0}(k+1),h_{1}(k+1)\\right] \\left[\n",
384 "\\begin{array}\n",
385 "[c]{c}%\n",
386 "w_{y,0}\\\\\n",
387 "w_{y,1}%\n",
388 "\\end{array}\n",
389 "\\right] \n",
390 "\\end{align*}\n"
391 ]
392 },
393 {
394 "cell_type": "markdown",
395 "metadata": {
396 "id": "MAvXBZodx8RP"
397 },
398 "source": [
399 "We compute this for $k=1,2,3$ and compare with the output of the model:"
400 ]
401 },
402 {
403 "cell_type": "code",
404 "metadata": {
405 "id": "kMOhAImzL7Ch"
406 },
407 "source": [
408 "w = demo_model.get_weights()\n",
409 "wx = w[0]\n",
410 "wh = w[1]\n",
411 "bh = w[2]\n",
412 "wy = w[3]\n",
413 "by = w[4]\n",
414 "x = np.array([1, 2, 3])\n",
415 "# Reshape the input to the required sample_size x time_steps x features \n",
416 "x_input = np.reshape(x,(1, 3, 1))\n",
417 "y_pred_model = demo_model.predict(x_input)\n",
418 "\n",
419 "\n",
420 "m = 2\n",
421 "h0 = np.zeros(m)\n",
422 "h1 = np.dot(x[0], wx) + np.dot(h0,wh) + bh\n",
423 "h2 = np.dot(x[1], wx) + np.dot(h1,wh) + bh\n",
424 "h3 = np.dot(x[2], wx) + np.dot(h2,wh) + bh\n",
425 "o3 = np.dot(h3, wy) + by\n",
426 "\n",
427 "print('h1 = ', h1,'h2 = ', h2,'h3 = ', h3)\n",
428 "\n",
429 "print(\"Prediction from network \", y_pred_model)\n",
430 "print(\"Prediction from our computation \", o3)"
431 ],
432 "execution_count": null,
433 "outputs": []
434 },
435 {
436 "cell_type": "code",
437 "source": [
438 "# the same using arrays\n",
439 "demo_model = create_RNN_functional(hidden_units=2, dense_units=1, input_shape=(3,1), \n",
440 " activation=['linear', 'linear'],return_sequences=True)\n",
441 "w = demo_model.get_weights()\n",
442 "\n",
443 "x = np.array([1, 2, 3])\n",
444 "# Reshape the input to the required sample_size x time_steps x features \n",
445 "x_input = np.reshape(x,(1, 3, 1))\n",
446 "y_pred_model = demo_model.predict(x_input)\n",
447 "\n",
448 "h = np.zeros(2)\n",
449 "o = np.empty(3)\n",
450 "for i in range(3):\n",
451 " h = np.dot(x[i], w[0]) + np.dot(h, w[1]) + w[2]\n",
452 " o[i]=np.dot(h, w[3]) + w[4]\n",
453 "\n",
454 "print(\"Prediction from network \", y_pred_model)\n",
455 "print(\"Prediction from our computation \", o)"
456 ],
457 "metadata": {
458 "id": "Q2kuhi2KBNY4"
459 },
460 "execution_count": null,
461 "outputs": []
462 },
463 {
464 "cell_type": "markdown",
465 "source": [
466 "### Stateful model\n",
467 "\n",
468 "In a stateful model, when the model is called multiple times, the model object remembers that it was called before and its final hidden state, then it starts from that hidden state in the next call. \n",
469 "\n",
470 "Note the model remembers only the hidden state from the last timestep not the sequence of hidden states for all timesteps, even if the model was built with `return_sequences=True`. The hidden states from all timesteps except the last one are lost. "
471 ],
472 "metadata": {
473 "id": "8DLr5jfaKc8E"
474 }
475 },
476 {
477 "cell_type": "code",
478 "source": [
479 "# stateful model\n",
480 "demo_model = create_RNN_functional(hidden_units=2, dense_units=1, \n",
481 " activation=['linear', 'linear'],return_sequences=True,\n",
482 " stateful=True,batch_shape=(1,3,1))\n",
483 "print(demo_model.summary())\n",
484 "from keras.utils.vis_utils import plot_model\n",
485 "plot_model(demo_model, to_file='model_plot.png', \n",
486 " show_shapes=True, show_layer_names=True)\n",
487 "\n",
488 "w = demo_model.get_weights()\n",
489 "\n",
490 "x = np.array([1, 2, 3])\n",
491 "# Reshape the input to the required batch_size x time_steps x features \n",
492 "x_input = np.reshape(x,(1, 3, 1))\n",
493 "y_pred_model1 = demo_model.predict(x_input)\n",
494 "y_pred_model2 = demo_model.predict(x_input)\n",
495 "y_pred_model3 = demo_model.predict(x_input)\n",
496 "\n",
497 "# batch 1\n",
498 "h = np.zeros(2)\n",
499 "o1 = np.empty(3)\n",
500 "for i in range(3):\n",
501 " h = np.dot(x[i], w[0]) + np.dot(h, w[1]) + w[2]\n",
502 " o1[i]=np.dot(h, w[3]) + w[4]\n",
503 "\n",
504 "# batch 2\n",
505 "# we do not zero out h - stateful model remembers h \n",
506 "o2 = np.empty(3)\n",
507 "for i in range(3):\n",
508 " h = np.dot(x[i], w[0]) + np.dot(h, w[1]) + w[2]\n",
509 " o2[i]=np.dot(h, w[3]) + w[4]\n",
510 "\n",
511 "# batch 3\n",
512 "# we do not zero out h - stateful model remembers h \n",
513 "o3 = np.empty(3)\n",
514 "for i in range(3):\n",
515 " h = np.dot(x[i], w[0]) + np.dot(h, w[1]) + w[2]\n",
516 " o3[i]=np.dot(h, w[3]) + w[4]\n",
517 "\n",
518 "print(\"Prediction from network 1\",y_pred_model1)\n",
519 "print(\"Prediction from network 2\",y_pred_model2)\n",
520 "print(\"Prediction from network 3\",y_pred_model3)\n",
521 "print(\"Prediction from our computation 1\", o1)\n",
522 "print(\"Prediction from our computation 2\", o2)\n",
523 "print(\"Prediction from our computation 2\", o3)"
524 ],
525 "metadata": {
526 "id": "SLYRVhFGHa4r"
527 },
528 "execution_count": null,
529 "outputs": []
530 },
531 {
532 "cell_type": "code",
533 "source": [
534 "# stateful model with batch consisting of one sample\n",
535 "# rewritten as loop over model calls\n",
536 "demo_model = create_RNN_functional(hidden_units=2, dense_units=1, \n",
537 " activation=['linear', 'linear'],return_sequences=True,\n",
538 " stateful=True,batch_shape=(1,3,1))\n",
539 "print(demo_model.summary())\n",
540 "from keras.utils.vis_utils import plot_model\n",
541 "plot_model(demo_model, to_file='model_plot.png', \n",
542 " show_shapes=True, show_layer_names=True)\n",
543 "\n",
544 "w = demo_model.get_weights()\n",
545 "\n",
546 "x = np.array([1, 2, 3])\n",
547 "# Reshape the input to the required batch_size x time_steps x features \n",
548 "x_input = np.reshape(x,(1, 3, 1))\n",
549 "y_pred_model1 = demo_model.predict(x_input)\n",
550 "y_pred_model2 = demo_model.predict(x_input)\n",
551 "\n",
552 "# batch 1\n",
553 "h = np.zeros(2)\n",
554 "o = np.empty((2,3))\n",
555 "for j in range(2): # loop over batches\n",
556 " # only one sample in batch\n",
557 " for i in range(3): # loop over timesteps\n",
558 " h = np.dot(x[i], w[0]) + np.dot(h, w[1]) + w[2]\n",
559 " o[j,i]=np.dot(h, w[3]) + w[4]\n",
560 "\n",
561 "print(\"Prediction from network 1\",y_pred_model1)\n",
562 "print(\"Prediction from network 2\",y_pred_model2)\n",
563 "print(\"Prediction from our computation\", o)"
564 ],
565 "metadata": {
566 "colab": {
567 "base_uri": "https://localhost:8080/"
568 },
569 "id": "lSg-HiqAQVWI",
570 "outputId": "fe0bb01b-40e8-4afd-ad48-3a1f55b106cb"
571 },
572 "execution_count": 46,
573 "outputs": [
574 {
575 "output_type": "stream",
576 "name": "stdout",
577 "text": [
578 "Model: \"model_13\"\n",
579 "_________________________________________________________________\n",
580 " Layer (type) Output Shape Param # \n",
581 "=================================================================\n",
582 " input_14 (InputLayer) [(1, 3, 1)] 0 \n",
583 " \n",
584 " simple_rnn_13 (SimpleRNN) (1, 3, 2) 8 \n",
585 " \n",
586 " dense_13 (Dense) (1, 3, 1) 3 \n",
587 " \n",
588 "=================================================================\n",
589 "Total params: 11\n",
590 "Trainable params: 11\n",
591 "Non-trainable params: 0\n",
592 "_________________________________________________________________\n",
593 "None\n",
594 "Prediction from network 1 [[[0.5394513 ]\n",
595 " [0.81535405]\n",
596 " [0.9701195 ]]]\n",
597 "Prediction from network 2 [[[-0.03764684]\n",
598 " [ 0.761226 ]\n",
599 " [ 1.6331315 ]]]\n",
600 "Prediction from our computation [[ 0.53945129 0.81535404 0.97011949]\n",
601 " [-0.03764684 0.76122602 1.63313155]]\n"
602 ]
603 }
604 ]
605 },
606 {
607 "cell_type": "markdown",
608 "metadata": {
609 "id": "JopR12ZaVAyS"
610 },
611 "source": [
612 "The predictions came out the same! This confirms that we know what the network is doing."
613 ]
614 },
615 {
616 "cell_type": "markdown",
617 "metadata": {
618 "id": "dHGF-tofMpJP"
619 },
620 "source": [
621 "## Step 1, 2: Reading Data and Splitting Into Train And Test"
622 ]
623 },
624 {
625 "cell_type": "markdown",
626 "metadata": {
627 "id": "5x748YZuY-yL"
628 },
629 "source": [
630 "The following function reads the train and test data from a given URL and splits it into a given percentage of train and test data. It returns single dimensional arrays for train and test data after scaling the data between 0 and 1 using MinMaxScaler from scikit-learn."
631 ]
632 },
633 {
634 "cell_type": "code",
635 "metadata": {
636 "id": "JyrxUuiuL8gv"
637 },
638 "source": [
639 "# Parameter split_percent defines the ratio of training examples\n",
640 "def get_train_test(data, split_percent=0.8):\n",
641 " scaler = MinMaxScaler(feature_range=(0, 1))\n",
642 " data = scaler.fit_transform(data).flatten()\n",
643 " n = len(data)\n",
644 " # Point for splitting data into train and test\n",
645 " split = int(n*split_percent)\n",
646 " train_data = data[range(split)]\n",
647 " test_data = data[split:]\n",
648 " return train_data, test_data, data\n",
649 "\n",
650 "sunspots_url = 'https://raw.githubusercontent.com/jbrownlee/Datasets/master/monthly-sunspots.csv'\n",
651 "df = read_csv(sunspots_url, usecols=[1], engine='python')\n",
652 "train_data, test_data, data = get_train_test(np.array(df.values.astype('float32')))"
653 ],
654 "execution_count": null,
655 "outputs": []
656 },
657 {
658 "cell_type": "markdown",
659 "metadata": {
660 "id": "iCsHwJOcZMJ7"
661 },
662 "source": [
663 "Let's print the data shape so that we know what we got."
664 ]
665 },
666 {
667 "cell_type": "code",
668 "metadata": {
669 "id": "h5AmHug8JViT"
670 },
671 "source": [
672 "data.shape"
673 ],
674 "execution_count": null,
675 "outputs": []
676 },
677 {
678 "cell_type": "markdown",
679 "metadata": {
680 "id": "QHoBV8CSMt44"
681 },
682 "source": [
683 "## Step 3: Reshaping Data For Keras"
684 ]
685 },
686 {
687 "cell_type": "markdown",
688 "metadata": {
689 "id": "B1CW_mu8Zbwb"
690 },
691 "source": [
692 "The next step is to prepare the data for Keras model training. The input array should be shaped as: **(total_samples, x time_steps, x features)**.\n",
693 "There are many ways of preparing time series data for training. We’ll create input rows with non-overlapping time steps. An example is shown in the figure below. Here time_steps denotes the number of previous time steps to use for predicting the next value of the time series data. We have for time_steps = 2, features = 1, and the first 6 terms are split total_samples=3 samples: 0, 10 predict the next term 20, then 20, 30 predict the next term 40, etc."
694 ]
695 },
696 {
697 "cell_type": "markdown",
698 "metadata": {
699 "id": "OeEc_dmqZmtx"
700 },
701 "source": [
702 "<img src=\"https://machinelearningmastery.com/wp-content/uploads/2021/09/rnnCode2.png\">"
703 ]
704 },
705 {
706 "cell_type": "markdown",
707 "metadata": {
708 "id": "iLqm8291Pd5X"
709 },
710 "source": [
711 "The following function get_XY() takes a one dimensional array as input and converts it to the required input X and target Y arrays."
712 ]
713 },
714 {
715 "cell_type": "code",
716 "metadata": {
717 "id": "IxJEj52BL__o"
718 },
719 "source": [
720 "# Prepare the input X and target Y\n",
721 "def get_XY(dat, time_steps):\n",
722 " # Indices of target array\n",
723 " Y_ind = np.arange(time_steps, len(dat), time_steps)\n",
724 " Y = dat[Y_ind]\n",
725 " # Prepare X\n",
726 " rows_x = len(Y)\n",
727 " X = dat[range(time_steps*rows_x)]\n",
728 " X = np.reshape(X, (rows_x, time_steps, 1)) \n",
729 " return X, Y"
730 ],
731 "execution_count": null,
732 "outputs": []
733 },
734 {
735 "cell_type": "markdown",
736 "metadata": {
737 "id": "RFhadJjzQO7p"
738 },
739 "source": [
740 "For illustration, on the simple example above it returns the expected result: "
741 ]
742 },
743 {
744 "cell_type": "code",
745 "metadata": {
746 "id": "V38oXJ32QiFK"
747 },
748 "source": [
749 "dat = np.linspace(0.,70.,8).reshape(-1,1)\n",
750 "print(\"dat shape=\",dat.shape)\n",
751 "X, Y = get_XY(dat, 2)\n",
752 "print(\"X shape=\",X.shape)\n",
753 "print(\"Y shape=\",Y.shape)\n",
754 "#print('dat=',dat)\n",
755 "print('X=',X)\n",
756 "print('Y=',Y)\n"
757 ],
758 "execution_count": null,
759 "outputs": []
760 },
761 {
762 "cell_type": "markdown",
763 "metadata": {
764 "id": "4V4IE7TvQpDW"
765 },
766 "source": [
767 "Now use it for the sunspot data. We’ll use 12 time_steps for the sunspots dataset as the sunspots generally have a cycle of 12 months. You can experiment with other values of time_steps."
768 ]
769 },
770 {
771 "cell_type": "code",
772 "metadata": {
773 "id": "YBnBPDxiQsjL"
774 },
775 "source": [
776 "time_steps = 24\n",
777 "trainX, trainY = get_XY(train_data, time_steps)\n",
778 "testX, testY = get_XY(test_data, time_steps)\n",
779 "print(\"trainX shape=\",trainX.shape)\n",
780 "print(\"trainY shape=\",trainY.shape)\n",
781 "print(\"testX shape=\",testX.shape)\n",
782 "print(\"testY shape=\",testY.shape)"
783 ],
784 "execution_count": null,
785 "outputs": []
786 },
787 {
788 "cell_type": "markdown",
789 "metadata": {
790 "id": "Xz2JRTGKMzo2"
791 },
792 "source": [
793 "## Step 4: Create RNN Model And Train"
794 ]
795 },
796 {
797 "cell_type": "code",
798 "metadata": {
799 "id": "SyAE6XLnMGDO"
800 },
801 "source": [
802 "model = create_RNN(hidden_units=3, dense_units=1, input_shape=(time_steps,1), \n",
803 " activation=['tanh', 'tanh'])\n",
804 "model.fit(trainX, trainY, epochs=20, batch_size=1, verbose=2)"
805 ],
806 "execution_count": null,
807 "outputs": []
808 },
809 {
810 "cell_type": "markdown",
811 "metadata": {
812 "id": "tluiPIaxM9FH"
813 },
814 "source": [
815 "## Step 5: Compute And Print The Root Mean Square Error"
816 ]
817 },
818 {
819 "cell_type": "code",
820 "metadata": {
821 "id": "LnWdCmqJMK-k"
822 },
823 "source": [
824 "def print_error(trainY, testY, train_predict, test_predict): \n",
825 " # Error of predictions\n",
826 " train_rmse = math.sqrt(mean_squared_error(trainY, train_predict))\n",
827 " test_rmse = math.sqrt(mean_squared_error(testY, test_predict))\n",
828 " # Print RMSE\n",
829 " print('Train RMSE: %.3f RMSE' % (train_rmse))\n",
830 " print('Test RMSE: %.3f RMSE' % (test_rmse)) \n",
831 "\n",
832 "# make predictions\n",
833 "train_predict = model.predict(trainX)\n",
834 "test_predict = model.predict(testX)\n",
835 "# Mean square error\n",
836 "print_error(trainY, testY, train_predict, test_predict)"
837 ],
838 "execution_count": null,
839 "outputs": []
840 },
841 {
842 "cell_type": "markdown",
843 "metadata": {
844 "id": "eed7244NNHs9"
845 },
846 "source": [
847 ""
848 ]
849 },
850 {
851 "cell_type": "markdown",
852 "metadata": {
853 "id": "nDwJvflyNGz3"
854 },
855 "source": [
856 "## Step 6: View The result"
857 ]
858 },
859 {
860 "cell_type": "code",
861 "metadata": {
862 "id": "fQPjWHWvMMGL"
863 },
864 "source": [
865 "# Plot the result\n",
866 "def plot_result(trainY, testY, train_predict, test_predict):\n",
867 " actual = np.append(trainY, testY)\n",
868 " predictions = np.append(train_predict, test_predict)\n",
869 " rows = len(actual)\n",
870 " plt.figure(figsize=(15, 6), dpi=80)\n",
871 " plt.plot(range(rows), actual)\n",
872 " plt.plot(range(rows), predictions)\n",
873 " plt.axvline(x=len(trainY), color='r')\n",
874 " plt.legend(['Actual', 'Predictions'])\n",
875 " plt.xlabel('Observation number after given time steps')\n",
876 " plt.ylabel('Sunspots scaled')\n",
877 " plt.title('Actual and Predicted Values. The Red Line Separates The Training And Test Examples')\n",
878 "plot_result(trainY, testY, train_predict, test_predict)"
879 ],
880 "execution_count": null,
881 "outputs": []
882 }
883 ]
884 }