| blob | c3ded6c82bf61a47c20e3f1c2e340ba4e2291634 |
1 {
2 "cells": [
3 {
4 "cell_type": "markdown",
5 "id": "a9df30a4-c9b1-4bcb-8183-1030ff1c1332",
6 "metadata": {},
7 "source": [
8 "# Tinker with the RNN\n",
9 "\n",
10 "The point of this notebook is to analyze the Physics-Initiated RNN on a granual scale. In each major section, we will present an issue with the RNN outputs and analyze ways to fix."
11 ]
12 },
13 {
14 "cell_type": "markdown",
15 "id": "41b6c912-a46b-461f-bde3-9add77ef0af6",
16 "metadata": {},
17 "source": [
18 "## Setup"
19 ]
20 },
21 {
22 "cell_type": "code",
23 "execution_count": null,
24 "id": "f1852edc-a583-4371-9bc8-c802ae6575ae",
25 "metadata": {},
26 "outputs": [],
27 "source": [
28 "# Setup\n",
29 "import reproducibility\n",
30 "\n",
31 "# both can change\n",
32 "# Environment\n",
33 "import numpy as np\n",
34 "import pandas as pd\n",
35 "import tensorflow as tf\n",
36 "from keras.models import Sequential\n",
37 "from keras.layers import Dense, SimpleRNN\n",
38 "# from keras.utils.vis_utils import plot_model\n",
39 "from keras.utils import plot_model\n",
40 "\n",
41 "from sklearn.preprocessing import MinMaxScaler\n",
42 "from sklearn.metrics import mean_squared_error\n",
43 "import math\n",
44 "import json\n",
45 "import matplotlib.pyplot as plt\n",
46 "import tensorflow as tf\n",
47 "import keras.backend as K\n",
48 "# from keras.utils.vis_utils import plot_model\n",
49 "from scipy.interpolate import LinearNDInterpolator, interpn\n",
50 "from scipy.optimize import root\n",
51 "import pickle, os\n",
52 "from utils import hash2\n",
53 "%matplotlib inline\n",
54 "\n",
55 "# Local modules for handling data and running moisture models\n",
56 "import data_funcs as datf\n",
57 "from data_funcs import format_raws, retrieve_raws, format_precip, fixnan, load_and_fix_data\n",
58 "from data_funcs import raws_data, synthetic_data, plot_data, check_data, rmse_data, to_json, from_json\n",
59 "import moisture_models as mod\n",
60 "from moisture_rnn import run_case, run_rnn, create_RNN_2, staircase, create_rnn_data, train_rnn, rnn_predict\n"
61 ]
62 },
63 {
64 "cell_type": "code",
65 "execution_count": null,
66 "id": "d972a09e-205c-4a8d-bad5-6ba3f6ed6681",
67 "metadata": {},
68 "outputs": [],
69 "source": [
70 "# Data\n",
71 "# Change directory for data read/write\n",
72 "\n",
73 "dict_file='data/raws_CO_202306.pickle' # input path of FMDA dictionaries\n",
74 "reproducibility_file='data/reproducibility_dict.pickle'\n",
75 "\n",
76 "# read test datasets\n",
77 "test_dict={}\n",
78 "test_dict.update(load_and_fix_data(dict_file))\n",
79 "print(test_dict.keys())\n",
80 "\n",
81 "repro_dict={}\n",
82 "repro_dict.update(load_and_fix_data(reproducibility_file))\n",
83 "print(repro_dict.keys())"
84 ]
85 },
86 {
87 "cell_type": "markdown",
88 "id": "171cb81b-f8b8-4e75-8e12-09daa87f7d01",
89 "metadata": {},
90 "source": [
91 "## Poor initial fits \n",
92 "\n",
93 "Sometimes the RNN has very poor initial fits. When the activation is linear, this results in largely overestimating the FM. We will examine whether this is an issue with the data or with fitting the initial weights.\n",
94 "\n",
95 "* Case studies for overestimating: `CPTC2_202306010000`, `CHAC2_202306010000`, `CHRC2_202306010000`, `WLCC2_202306010000`"
96 ]
97 },
98 {
99 "cell_type": "code",
100 "execution_count": null,
101 "id": "c00cbe6f-6f56-4af3-97c8-2ef3306458d5",
102 "metadata": {},
103 "outputs": [],
104 "source": [
105 "# Build Case Data\n",
106 "id = \"CPTC2_202306010000\"\n",
107 "case_data=test_dict[id]\n",
108 "case_data[\"hours\"]=len(case_data['fm'])\n",
109 "case_data[\"h2\"]=int(24*20)"
110 ]
111 },
112 {
113 "cell_type": "markdown",
114 "id": "1ea15a94-5296-4134-9d1e-13c68301c726",
115 "metadata": {},
116 "source": [
117 "First, we check how the Augmented KF performs."
118 ]
119 },
120 {
121 "cell_type": "code",
122 "execution_count": null,
123 "id": "987cfd43-5cb0-410d-8835-9468503b06fc",
124 "metadata": {},
125 "outputs": [],
126 "source": [
127 "m, Ec =mod.run_augmented_kf(case_data)\n",
128 "case_data[\"m\"] = m\n",
129 "plot_data(case_data,title2='augmented KF')"
130 ]
131 },
132 {
133 "cell_type": "markdown",
134 "id": "a8fb6618-e48c-40af-810d-8206bee1d078",
135 "metadata": {},
136 "source": [
137 "The Augmented KF is reasonable."
138 ]
139 },
140 {
141 "cell_type": "markdown",
142 "id": "0bd36b36-0aee-4849-b049-2b66dad2f64c",
143 "metadata": {},
144 "source": [
145 "### Linear Activation overestimates\n",
146 "\n",
147 "Next, we initialize the RNN with linear activation and 1 single hidden unit and plot."
148 ]
149 },
150 {
151 "cell_type": "code",
152 "execution_count": null,
153 "id": "6345e4dc-fa6b-49b3-9840-86f84cfc1ca9",
154 "metadata": {},
155 "outputs": [],
156 "source": [
157 "# Linear Activation Params\n",
158 "params={'id':1,\n",
159 " 'purpose':'test 1',\n",
160 " 'cases':'all',\n",
161 " 'scale':1, # every feature in [0, scale]\n",
162 " 'rain_do':True,\n",
163 " 'verbose':False,\n",
164 " 'timesteps':5,\n",
165 " 'activation':['linear','linear'],\n",
166 " 'hidden_units':1, \n",
167 " 'dense_units':1, # do not change\n",
168 " 'dense_layers':1, # do not change\n",
169 " 'centering':[0.0,0.0], # should be activation at 0\n",
170 " 'DeltaE':[0,-1], # bias correction\n",
171 " 'synthetic':False, # run also synthetic cases\n",
172 " 'T1': 0.1, # 1/fuel class (10)\n",
173 " 'fm_raise_vs_rain': 2.0, # fm increase per mm rain \n",
174 " 'train_frac':0.5, # time fraction to spend on training\n",
175 " 'epochs':1000,\n",
176 " 'verbose_fit':False,\n",
177 " 'verbose_weights':False,\n",
178 "}"
179 ]
180 },
181 {
182 "cell_type": "code",
183 "execution_count": null,
184 "id": "3f98aee6-7118-412f-a1ac-de2116608b08",
185 "metadata": {},
186 "outputs": [],
187 "source": [
188 "reproducibility.set_seed() # Set seed for reproducibility\n",
189 "rnn_dat = create_rnn_data(case_data,params)\n",
190 "model1 = train_rnn(\n",
191 " rnn_dat,\n",
192 " params,\n",
193 " rnn_dat['hours'],\n",
194 " fit=False\n",
195 ")\n",
196 "case_data['m'] = rnn_predict(model1, params, rnn_dat)\n",
197 "fit_linear = case_data['m']"
198 ]
199 },
200 {
201 "cell_type": "code",
202 "execution_count": null,
203 "id": "b1f7826b-b951-4cf3-9626-49ff0a84a53b",
204 "metadata": {},
205 "outputs": [],
206 "source": [
207 "plot_data(case_data,title2='Initial RNN Linear')"
208 ]
209 },
210 {
211 "cell_type": "code",
212 "execution_count": null,
213 "id": "06de0b51-4e0b-44fc-95d7-269217f5f21f",
214 "metadata": {},
215 "outputs": [],
216 "source": [
217 "model1.get_weights()"
218 ]
219 },
220 {
221 "cell_type": "markdown",
222 "id": "504dc3ad-6ff2-4d4d-afe5-20101e49018b",
223 "metadata": {},
224 "source": [
225 "Try again with different `fm_raise_vs_rain`..."
226 ]
227 },
228 {
229 "cell_type": "code",
230 "execution_count": null,
231 "id": "0f30c02c-cedc-46be-98a4-4a4dcda57657",
232 "metadata": {},
233 "outputs": [],
234 "source": [
235 "# Linear Activation Params\n",
236 "params={'id':1,\n",
237 " 'purpose':'test 1',\n",
238 " 'cases':'all',\n",
239 " 'scale':1, # every feature in [0, scale]\n",
240 " 'rain_do':True,\n",
241 " 'verbose':False,\n",
242 " 'timesteps':5,\n",
243 " 'activation':['linear','linear'],\n",
244 " 'hidden_units':1, \n",
245 " 'dense_units':1, # do not change\n",
246 " 'dense_layers':1, # do not change\n",
247 " 'centering':[0.0,0.0], # should be activation at 0\n",
248 " 'DeltaE':[0,-1], # bias correction\n",
249 " 'synthetic':False, # run also synthetic cases\n",
250 " 'T1': 0.1, # 1/fuel class (10)\n",
251 " 'fm_raise_vs_rain': 0.1, # fm increase per mm rain \n",
252 " 'train_frac':0.5, # time fraction to spend on training\n",
253 " 'epochs':1000,\n",
254 " 'verbose_fit':False,\n",
255 " 'verbose_weights':False,\n",
256 "}"
257 ]
258 },
259 {
260 "cell_type": "code",
261 "execution_count": null,
262 "id": "0a9671b3-c506-4d30-b097-1092836c0124",
263 "metadata": {},
264 "outputs": [],
265 "source": [
266 "reproducibility.set_seed() # Set seed for reproducibility\n",
267 "rnn_dat = create_rnn_data(case_data,params)\n",
268 "model2 = train_rnn(\n",
269 " rnn_dat,\n",
270 " params,\n",
271 " rnn_dat['hours'],\n",
272 " fit=False\n",
273 ")\n",
274 "case_data['m'] = rnn_predict(model2, params, rnn_dat)\n",
275 "plot_data(case_data,title2='Initial RNN Linear')"
276 ]
277 },
278 {
279 "cell_type": "code",
280 "execution_count": null,
281 "id": "49026a00-f2e8-40b2-9421-12a14f498e39",
282 "metadata": {},
283 "outputs": [],
284 "source": [
285 "model2.get_weights()"
286 ]
287 },
288 {
289 "cell_type": "markdown",
290 "id": "f0668b47-5f65-4592-a38c-9e3245f0fcac",
291 "metadata": {},
292 "source": [
293 "The fit is much better. The only initial weight that changed is the 3rd weight in the input layer, which corresponds to rain.\n",
294 "\n",
295 "Note that .1 inches is about 2.5 mm, so this looks like a units issue.\n",
296 "\n",
297 "Documentation for MesoWest / Mesonet is inconsistent as to whether precipitation is in units of mm or inches:\n",
298 "* [Weather.gov](https://www.weather.gov/media/wrh/mesowest/MesoWest_Data_Variables_Definitions.pdf) says inches\n",
299 "* [Synoptic](https://developers.synopticdata.com/mesonet/v2/api-variables/) says mm.\n",
300 "\n",
301 "We investiage the input data and compare rainfall to historical accounts to see if we can deduce inches vs mm:"
302 ]
303 },
304 {
305 "cell_type": "code",
306 "execution_count": null,
307 "id": "966213db-6b2f-425d-9318-2ae5967fb499",
308 "metadata": {},
309 "outputs": [],
310 "source": [
311 "print(f\"RAWS Site {case_data['STID']} lat/lon: {case_data['lat']},{case_data['lon']}\")"
312 ]
313 },
314 {
315 "cell_type": "code",
316 "execution_count": null,
317 "id": "9cd458df-fcbd-4ad0-80f1-7525a85f7072",
318 "metadata": {},
319 "outputs": [],
320 "source": [
321 "# Get index of largest rainfall and plot series about it with times\n",
322 "ind = case_data['rain'].argmax()\n",
323 "x=case_data['rain'][np.max([ind-10, 0]) : np.min([ind+10, len(case_data['rain'])])]\n",
324 "dates = case_data['time'][np.max([ind-10, 0]) : np.min([ind+10, len(case_data['rain'])])]\n",
325 "plt.plot(dates, x, 'o-')\n",
326 "plt.xticks(rotation=90, ha='right')\n",
327 "print(x)\n",
328 "print(dates)"
329 ]
330 },
331 {
332 "cell_type": "markdown",
333 "id": "5cc38b95-065a-4f82-b0cd-c3c4267cf1b5",
334 "metadata": {},
335 "source": [
336 "A quick historical weather search says the max rainfall at this location was roughly 5mm: https://open-meteo.com/en/docs/historical-weather-api#start_date=2023-06-14&end_date=2023-06-15&hourly=rain&daily=&timezone=GMT\n",
337 "\n",
338 "Investigating the output from `MesoPy` (from notebook create_RAWS_dict.ipynb), it says the units are mm. So units might not be the issue."
339 ]
340 },
341 {
342 "cell_type": "markdown",
343 "id": "0e1581a2-1beb-41dc-ad0e-7fe0e9d57378",
344 "metadata": {},
345 "source": [
346 "The initial weight is set by multiplying the parameter of fm increase per mm rain by the scaled max fm divided by the scaled max rain, so maybe this could be address with scaling."
347 ]
348 },
349 {
350 "cell_type": "markdown",
351 "id": "5e207728-8eec-4a77-bd18-0b2c19f2deaa",
352 "metadata": {},
353 "source": [
354 "### Overestimate issues with other activations"
355 ]
356 },
357 {
358 "cell_type": "code",
359 "execution_count": null,
360 "id": "bdf319ea-817e-45b2-bf9d-8d4823970db1",
361 "metadata": {},
362 "outputs": [],
363 "source": [
364 "# Tanh Activation Params\n",
365 "params={'id':1,\n",
366 " 'purpose':'test 1',\n",
367 " 'cases':'all',\n",
368 " 'scale':1, # every feature in [0, scale]\n",
369 " 'rain_do':True,\n",
370 " 'verbose':False,\n",
371 " 'timesteps':5,\n",
372 " 'activation':['tanh','tanh'],\n",
373 " 'hidden_units':1, \n",
374 " 'dense_units':1, # do not change\n",
375 " 'dense_layers':1, # do not change\n",
376 " 'centering':[0.0,0.0], # should be activation at 0\n",
377 " 'DeltaE':[0,-1], # bias correction\n",
378 " 'synthetic':False, # run also synthetic cases\n",
379 " 'T1': 0.1, # 1/fuel class (10)\n",
380 " 'fm_raise_vs_rain': 2, # fm increase per mm rain \n",
381 " 'train_frac':0.5, # time fraction to spend on training\n",
382 " 'epochs':1000,\n",
383 " 'verbose_fit':False,\n",
384 " 'verbose_weights':False,\n",
385 "}"
386 ]
387 },
388 {
389 "cell_type": "code",
390 "execution_count": null,
391 "id": "22ab5251-f5a8-4fd1-8eec-75f5f17f95c6",
392 "metadata": {},
393 "outputs": [],
394 "source": [
395 "reproducibility.set_seed() # Set seed for reproducibility\n",
396 "rnn_dat = create_rnn_data(case_data,params)\n",
397 "model3 = train_rnn(\n",
398 " rnn_dat,\n",
399 " params,\n",
400 " rnn_dat['hours'],\n",
401 " fit=False\n",
402 ")\n",
403 "case_data['m'] = rnn_predict(model3, params, rnn_dat)\n",
404 "fit_tanh = case_data['m']\n",
405 "plot_data(case_data,title2='Initial RNN Tanh')"
406 ]
407 },
408 {
409 "cell_type": "code",
410 "execution_count": null,
411 "id": "47a2f0e5-b6cb-4c53-a49a-75715c89a9a2",
412 "metadata": {},
413 "outputs": [],
414 "source": [
415 "model3.get_weights()"
416 ]
417 },
418 {
419 "cell_type": "markdown",
420 "id": "f7a8f2d9-ed7e-4beb-b40e-51473e42f06b",
421 "metadata": {},
422 "source": [
423 "The overestimation is no longer present with only changing the activation. The initial weight for rain did not have the dramatic effect seen before.\n",
424 "\n",
425 "We will reproduce some of the calculations manually below to understand the issue. The hidden state at `t=0` is initialized to `0` (according to ChatGPT, Jan, and other internet sources)"
426 ]
427 },
428 {
429 "cell_type": "code",
430 "execution_count": null,
431 "id": "fcd830d8-bf30-4190-9efb-a789ffa9a0e2",
432 "metadata": {},
433 "outputs": [],
434 "source": [
435 "# First model value\n",
436 "print(\"~\"*50)\n",
437 "print(f\"Linear Model at t=0: {fit_linear[0]}\")\n",
438 "print(f\"Tanh Model at t=0: {fit_tanh[0]}\")"
439 ]
440 },
441 {
442 "cell_type": "code",
443 "execution_count": null,
444 "id": "1ac600e9-d652-4549-931f-cc5781b9e83c",
445 "metadata": {},
446 "outputs": [],
447 "source": [
448 "weights=model3.get_weights() # Same for both tanh and linear activation\n",
449 "data0 = rnn_dat['Et'][0] # Input data at time 0\n",
450 "\n",
451 "#print(data0[0]*weights[0][0] + data0[1]*weights[0][1] + data0[2]*weights[0][2]) # handcalc\n",
452 "#print(np.dot(data0, weights[0])) # with linalg \n",
453 "\n",
454 "# Dot prod is initial weights \n",
455 "# initial state is 0 times weight 1\n",
456 "# weight2 is 0 bias at RNN layer\n",
457 "# Add em up because of linear activation\n",
458 "# weights 3 is weight into the Dense layer, initialized at 1\n",
459 "# weight 4 is final bias\n",
460 "# Times scale at end\n",
461 "print(\n",
462 " f\"Linear: {((np.dot(data0, weights[0]) + 0*weights[1] + weights[2])*weights[3] + weights[4])*rnn_dat['scale_fm']}\"\n",
463 ") \n",
464 "\n",
465 "# Repeat with tanh, same steps except tanh function instead of adding (Linear)\n",
466 "foo = model3.get_layer(model3.layers[1].name).activation # extract activation function\n",
467 "\n",
468 "print(\n",
469 " f\"Tanh: {(foo(np.dot(data0, weights[0]) + 0*weights[1] + weights[2])*weights[3] + weights[4])*rnn_dat['scale_fm']}\"\n",
470 ") "
471 ]
472 },
473 {
474 "cell_type": "markdown",
475 "id": "40e83aa3-3251-493e-bb60-64088d417885",
476 "metadata": {},
477 "source": [
478 "The output is the same to about 7 decimal places.\n",
479 "\n",
480 "Next we show how blowup happens with linear activation. Currently, tanh activation feeds into a dense unit with weight 1, so for 1 initialized RNN cell the final dense layer does not change the output. The calculation above used the inputs at time zero, which had zero rain. Consider the same calculation for a range of rain values, from 0 to 6, the max observed in the case data"
481 ]
482 },
483 {
484 "cell_type": "code",
485 "execution_count": null,
486 "id": "784eec1f-db0f-4f82-9d40-98075da332be",
487 "metadata": {},
488 "outputs": [],
489 "source": [
490 "data0 = rnn_dat['Et'][0]\n",
491 "x = np.linspace(0, 6, 20)\n",
492 "y_lin = np.zeros(20)\n",
493 "y_tanh = np.zeros(20)\n",
494 "\n",
495 "for i in range(0, 20):\n",
496 " data0[2] = x[i]\n",
497 " y_lin[i]=((np.dot(data0, weights[0]) + 0*weights[1] + weights[2])*weights[3] + weights[4])*rnn_dat['scale_fm']\n",
498 " y_tanh[i]=(foo(np.dot(data0, weights[0]) + 0*weights[1] + weights[2])*weights[3] + weights[4])*rnn_dat['scale_fm']\n",
499 "\n",
500 "plt.plot(x, y_lin, '-o', label=\"Linear Activation\")\n",
501 "plt.plot(x, y_tanh, '-o', label=\"Tanh Activation\")\n",
502 "plt.title(\"Initial Values at t=0, varying rain at time 0\")\n",
503 "plt.legend()"
504 ]
505 },
506 {
507 "cell_type": "markdown",
508 "id": "70caf520-7032-4b15-9bbc-aa537374e952",
509 "metadata": {},
510 "source": [
511 "### Discussion of Initial Fit Overestimation\n",
512 "\n",
513 "Is 2 % fm increase per mm of rain simply too much? Is threshold rainfall accounted for? \n",
514 "\n",
515 "The tanh activation avoids blow up effect of rain, but not due to physically reasonable processes."
516 ]
517 },
518 {
519 "cell_type": "markdown",
520 "id": "b3cf7eea-bef8-4ffd-b0a1-c91c86579c6d",
521 "metadata": {},
522 "source": [
523 "### Sigmoid Underestimating in initial fit\n",
524 "\n",
525 "* Case study: `CPTC2_202306010000`, `CHRC2_202306010000` with sigmoid activation\n",
526 "\n",
527 "Note first was a case study for overfitting with linear activation."
528 ]
529 },
530 {
531 "cell_type": "code",
532 "execution_count": null,
533 "id": "e91f064a-1d3a-43c2-96e1-e5dd53ea0ef5",
534 "metadata": {},
535 "outputs": [],
536 "source": [
537 "# Sigmoid Activation Params\n",
538 "params={'id':1,\n",
539 " 'purpose':'test 1',\n",
540 " 'cases':'all',\n",
541 " 'scale':1, # every feature in [0, scale]\n",
542 " 'rain_do':True,\n",
543 " 'verbose':False,\n",
544 " 'timesteps':5,\n",
545 " 'activation':['sigmoid','sigmoid'],\n",
546 " 'hidden_units':1, \n",
547 " 'dense_units':1, # do not change\n",
548 " 'dense_layers':1, # do not change\n",
549 " 'centering':[0.5,0.5], # should be activation at 0\n",
550 " 'DeltaE':[0,-1], # bias correction\n",
551 " 'synthetic':False, # run also synthetic cases\n",
552 " 'T1': 0.1, # 1/fuel class (10)\n",
553 " 'fm_raise_vs_rain': 2, # fm increase per mm rain \n",
554 " 'train_frac':0.5, # time fraction to spend on training\n",
555 " 'epochs':1000,\n",
556 " 'verbose_fit':False,\n",
557 " 'verbose_weights':False,\n",
558 "}"
559 ]
560 },
561 {
562 "cell_type": "code",
563 "execution_count": null,
564 "id": "7da7068d-c6d6-4d1b-871b-a6276dfb3152",
565 "metadata": {},
566 "outputs": [],
567 "source": [
568 "reproducibility.set_seed() # Set seed for reproducibility\n",
569 "id = \"CPTC2_202306010000\"\n",
570 "case_data=test_dict[id]\n",
571 "rnn_dat = create_rnn_data(case_data,params)\n",
572 "model_predict = train_rnn(\n",
573 " rnn_dat,\n",
574 " params,\n",
575 " rnn_dat['hours'],\n",
576 " fit=False\n",
577 ")\n",
578 "case_data['m'] = rnn_predict(model_predict, params, rnn_dat)\n",
579 "plot_data(case_data,title2='Initial RNN Sigmoid')"
580 ]
581 },
582 {
583 "cell_type": "code",
584 "execution_count": null,
585 "id": "a9a190f1-f730-420c-84a2-3bb52dd86e4e",
586 "metadata": {},
587 "outputs": [],
588 "source": [
589 "model_predict.get_weights()"
590 ]
591 },
592 {
593 "cell_type": "markdown",
594 "id": "069661bf-31af-4794-a624-74f96b558fac",
595 "metadata": {},
596 "source": [
597 "The initial weights are the same as before except the final bias term, since that is affected by the centering needed to get the sigmoid function approximately linear on initialization."
598 ]
599 },
600 {
601 "cell_type": "markdown",
602 "id": "fc82bccc-3303-4f69-aaf0-5774026a7e11",
603 "metadata": {},
604 "source": [
605 "Decreasing centering..."
606 ]
607 },
608 {
609 "cell_type": "code",
610 "execution_count": null,
611 "id": "b332580c-9749-420a-b104-a5cc8367ba55",
612 "metadata": {},
613 "outputs": [],
614 "source": [
615 "# Sigmoid Activation Params\n",
616 "params={'id':1,\n",
617 " 'purpose':'test 1',\n",
618 " 'cases':'all',\n",
619 " 'scale':1, # every feature in [0, scale]\n",
620 " 'rain_do':True,\n",
621 " 'verbose':False,\n",
622 " 'timesteps':5,\n",
623 " 'activation':['sigmoid','sigmoid'],\n",
624 " 'hidden_units':1, \n",
625 " 'dense_units':1, # do not change\n",
626 " 'dense_layers':1, # do not change\n",
627 " 'centering':[.1,.1], # should be activation at 0\n",
628 " 'DeltaE':[0,-1], # bias correction\n",
629 " 'synthetic':False, # run also synthetic cases\n",
630 " 'T1': 0.1, # 1/fuel class (10)\n",
631 " 'fm_raise_vs_rain': 2, # fm increase per mm rain \n",
632 " 'train_frac':0.5, # time fraction to spend on training\n",
633 " 'epochs':1000,\n",
634 " 'verbose_fit':False,\n",
635 " 'verbose_weights':False,\n",
636 "}\n",
637 "model_predict = train_rnn(\n",
638 " rnn_dat,\n",
639 " params,\n",
640 " rnn_dat['hours'],\n",
641 " fit=False\n",
642 ")\n",
643 "case_data['m'] = rnn_predict(model_predict, params, rnn_dat)\n",
644 "plot_data(case_data,title2='Initial RNN Sigmoid')"
645 ]
646 },
647 {
648 "cell_type": "markdown",
649 "id": "024d5afa-6984-44fb-9f69-557a3506b4f2",
650 "metadata": {},
651 "source": [
652 "The initial fit is dominated by the rain events, and essentially flat everywhere else. The initialization for sigmoid is failing."
653 ]
654 },
655 {
656 "cell_type": "code",
657 "execution_count": null,
658 "id": "16692b30-2709-4b1d-b44a-589ae9f376ba",
659 "metadata": {},
660 "outputs": [],
661 "source": [
662 "# Manually work with activation function\n",
663 "\n",
664 "model_predict.layers[1].activation"
665 ]
666 },
667 {
668 "cell_type": "markdown",
669 "id": "a3207439-b193-4e32-9e27-8d18558c7cfd",
670 "metadata": {},
671 "source": [
672 "## Clipping Issues\n",
673 "\n",
674 "Some fitted RNN models \"clip\" the model, where values don't go above or below a certain threshold.\n",
675 "\n",
676 "* Clipped Below case study: `TT590_202306010000` with tanh activation\n",
677 "* Clipped Above case study: `CHRC2_202306010000`, `LKGC2_202306010000` with sigmoid\n",
678 "\n",
679 "Seems to correspond with underestimation initial fits."
680 ]
681 },
682 {
683 "cell_type": "code",
684 "execution_count": null,
685 "id": "1df5e15c-79d4-4f87-8d7d-da59e541d1d0",
686 "metadata": {},
687 "outputs": [],
688 "source": []
689 }
690 ],
691 "metadata": {
692 "kernelspec": {
693 "display_name": "Python 3 (ipykernel)",
694 "language": "python",
695 "name": "python3"
696 },
697 "language_info": {
698 "codemirror_mode": {
699 "name": "ipython",
700 "version": 3
701 },
702 "file_extension": ".py",
703 "mimetype": "text/x-python",
704 "name": "python",
705 "nbconvert_exporter": "python",
706 "pygments_lexer": "ipython3",
707 "version": "3.9.12"
708 }
709 },
710 "nbformat": 4,
711 "nbformat_minor": 5
712 }