| blob | d2bc914b80ed80eb576d042b58119e4c4d0aa00e |
1 {
2 "cells": [
3 {
4 "cell_type": "markdown",
5 "id": "7f39c039-5ee5-4b46-bf8f-1ae289db8d17",
6 "metadata": {},
7 "source": [
8 "# v2.3 run RNN Class with Spatial Training\n",
9 "\n",
10 "This notebook serves as a guide for using the RNN code in this project. It walks through the core functionality for the data pre-processing, setting up model hyperparameters, structuring data to feed into RNN, and evaluating prediction error with spatiotemporal cross-validation. "
11 ]
12 },
13 {
14 "cell_type": "markdown",
15 "id": "1e98fcc9-3079-45d1-aece-d656d70a4244",
16 "metadata": {},
17 "source": [
18 "## Setup\n",
19 "\n",
20 "We will import certain functions at code cells in relevant sections for clarity, but everything used will be included in this setup cell."
21 ]
22 },
23 {
24 "cell_type": "code",
25 "execution_count": 8,
26 "id": "31369263-1526-4117-b25d-c3ed71d298b0",
27 "metadata": {},
28 "outputs": [],
29 "source": [
30 "import numpy as np\n",
31 "from utils import print_dict_summary, print_first, str2time, logging_setup\n",
32 "import pickle\n",
33 "import logging\n",
34 "import os.path as osp\n",
35 "from moisture_rnn_pkl import pkl2train\n",
36 "from moisture_rnn import RNNParams, RNNData, RNN, rnn_data_wrap\n",
37 "from utils import hash2, read_yml, read_pkl, retrieve_url, Dict, print_dict_summary\n",
38 "from moisture_rnn import RNN\n",
39 "import reproducibility\n",
40 "from data_funcs import rmse, to_json, combine_nested, build_train_dict\n",
41 "from moisture_models import run_augmented_kf\n",
42 "import copy\n",
43 "import pandas as pd\n",
44 "import matplotlib.pyplot as plt\n",
45 "import yaml\n",
46 "import time"
47 ]
48 },
49 {
50 "cell_type": "code",
51 "execution_count": 6,
52 "id": "2c14b5e7-5f22-45d2-8bcd-73a94f9b25e8",
53 "metadata": {},
54 "outputs": [],
55 "source": [
56 "from IPython.display import Markdown, display\n",
57 "\n",
58 "# Helper function to make documentation a little prettier\n",
59 "def print_markdown_docstring(func):\n",
60 " display(Markdown(f\"```python\\n{func.__doc__}\\n```\"))"
61 ]
62 },
63 {
64 "cell_type": "markdown",
65 "id": "cede04ca-f1ae-411e-b014-e35493c8b9c9",
66 "metadata": {},
67 "source": [
68 "## Acquiring Data\n",
69 "\n",
70 "The expected format of the input data for this project is in the form of nested dictionaries with a particular structure. These dictionaries are produced by the process `build_fmda_dicts` within the `wrfxpy` branch `develop-72-jh`. These files are staged remotely as `pickle` files on the OpenWFM Demo site. The data consist of ground-based observations from RAWS stations and atmospheric data from the HRRR weather model interpolated to the location of the RAWS site. These data were collected by specifying a time period and a spatial bounding box, and all RAWS with FMC sensors were collected within those bounds and time frame.\n",
71 "\n",
72 "<mark>NOTE: as of 2024-10-22 the wrfxpy code is still needs to be merged with the latest changed from Angel. The code that makes fmda dictionaries shouldn't depend much on other changes within wrfxpy</mark>\n",
73 "\n",
74 "The first step is just to retrieve the files. The method is called `retrieve_url`, and lives in a python module `utils`. The `utils` functions are meant to apply to a general context, not anything specific to this project. It uses a method that calls `wget` as a subprocesses and saves to a target directory if the file doesn't already exist. You can force it to download with a function argument. The function documentation is printed below, then it is called using f-strings to make the code more concise."
75 ]
76 },
77 {
78 "cell_type": "code",
79 "execution_count": 5,
80 "id": "e32267c9-e5ef-475d-a5ec-00e7212996e4",
81 "metadata": {},
82 "outputs": [
83 {
84 "data": {
85 "text/markdown": [
86 "```python\n",
87 "\n",
88 " Downloads a file from a specified URL to a destination path.\n",
89 "\n",
90 " Parameters:\n",
91 " -----------\n",
92 " url : str\n",
93 " The URL from which to download the file.\n",
94 " dest_path : str\n",
95 " The destination path where the file should be saved.\n",
96 " force_download : bool, optional\n",
97 " If True, forces the download even if the file already exists at the destination path.\n",
98 " Default is False.\n",
99 "\n",
100 " Warnings:\n",
101 " ---------\n",
102 " Prints a warning if the file extension of the URL does not match the destination file extension.\n",
103 "\n",
104 " Raises:\n",
105 " -------\n",
106 " AssertionError:\n",
107 " If the download fails and the file does not exist at the destination path.\n",
108 "\n",
109 " Notes:\n",
110 " ------\n",
111 " This function uses the `wget` command-line tool to download the file. Ensure that `wget` is \n",
112 " installed and accessible from the system's PATH.\n",
113 "\n",
114 " Prints:\n",
115 " -------\n",
116 " A message indicating whether the file was downloaded or if it already exists at the \n",
117 " destination path.\n",
118 " \n",
119 "```"
120 ],
121 "text/plain": [
122 "<IPython.core.display.Markdown object>"
123 ]
124 },
125 "metadata": {},
126 "output_type": "display_data"
127 }
128 ],
129 "source": [
130 "print_markdown_docstring(retrieve_url)"
131 ]
132 },
133 {
134 "cell_type": "code",
135 "execution_count": 7,
136 "id": "aa00ee6d-1d13-46fc-a942-64578dfe5b7d",
137 "metadata": {},
138 "outputs": [
139 {
140 "name": "stdout",
141 "output_type": "stream",
142 "text": [
143 "Target data already exists at data/fmda_rocky_202403-05_f05.pkl\n"
144 ]
145 }
146 ],
147 "source": [
148 "filename = \"fmda_rocky_202403-05_f05.pkl\"\n",
149 "retrieve_url(\n",
150 " url = f\"https://demo.openwfm.org/web/data/fmda/dicts/{filename}\", \n",
151 " dest_path = f\"data/{filename}\")"
152 ]
153 },
154 {
155 "cell_type": "markdown",
156 "id": "8ced2117-f36b-427b-86ae-0dca1e9cdccf",
157 "metadata": {},
158 "source": [
159 "### Exploring the Nested Dictionary Structure \n",
160 "\n",
161 "The data dictionaries have the following structure:\n",
162 "\n",
163 "* Top level keys are RAWS station IDs and some additional string related to the time period.\n",
164 "* For each of the RAWS sites, there are 3 subdictionaries consisting of different types of data that pertain to that location.\n",
165 " - A `loc` subdirectory that consists of static information about the physical location of the RAWS site. This includes station ID name, longitude, latitude, elevation, and two grid coordinates named \"pixel_x\" and \"pixel_y\" <mark>This will be renamed to \"grid_coordinate\" in the future</mark>. These correspond to the transformation of the lon/lat coordinates from the RAWS site onto the regular HRRR grid.\n",
166 " - A `RAWS` subdirectory that includes at least FMC observations and the associated times returned by Synoptic. These times may not line up perfectly with the requested regular hours. In addition to the FMC data, any available ground-based sensor data for variables relevant to FMC were collected. These data are intended to be used as validation for the accuracy of the interpolated HRRR data.\n",
167 " - A `HRRR` subdirectory that includes atmospheric variables relevant to FMC. The formatted table below shows the variables used by this project, where band numbers come from [NOAA documentation](https://www.nco.ncep.noaa.gov/pmb/products/hrrr/hrrr.t00z.wrfprsf00.grib2.shtml). <mark>More variables will be collected in the future</mark>. The HRRR subdirectory is organized into forecast hours. Each forecast hour subdirectory should have all the same information, just at different times from the HRRR forecast. "
168 ]
169 },
170 {
171 "cell_type": "code",
172 "execution_count": 16,
173 "id": "8bebd077-2690-4b91-8779-a1223a5c91dc",
174 "metadata": {},
175 "outputs": [
176 {
177 "name": "stdout",
178 "output_type": "stream",
179 "text": [
180 "loading file data/fmda_rocky_202403-05_f05.pkl\n"
181 ]
182 },
183 {
184 "data": {
185 "text/plain": [
186 "dict_keys(['CPTC2_202403', 'CHAC2_202403', 'CHRC2_202403', 'DYKC2_202403', 'LKGC2_202403', 'CCEC2_202403', 'RDKC2_202403', 'RFRC2_202403', 'SAWC2_202403', 'WLCC2_202403', 'CCRU1_202403', 'HSRU1_202403', 'YLSU1_202403', 'BRLW4_202403', 'GTGW4_202403', 'SAWW4_202403', 'SPKW4_202403', 'ESPC2_202403', 'MRFC2_202403', 'PKLC2_202403', 'BRAU1_202403', 'NLPU1_202403', 'TS010_202403', 'CUHC2_202403', 'BAWC2_202403', 'BTAC2_202403', 'SOPC2_202403', 'BMOC2_202403', 'CYNC2_202403', 'TR223_202403', 'TCTM8_202403', 'TR337_202403', 'VLRW4_202403', 'TR383_202403', 'LEIW4_202403', 'TR390_202403', 'TS001_202403', 'HSYN1_202403', 'HRSN1_202403', 'SBFN1_202403', 'DOHS2_202403', 'BKFS2_202403', 'CRRS2_202403', 'NMOS2_202403', 'RDCS2_202403', 'TGSK1_202403', 'QNRK1_202403', 'RESN1_202403', 'VRFN1_202403', 'KSHC2_202403', 'TR563_202403', 'CGLK1_202403', 'MRLS2_202403', 'TR755_202403', 'DEOI4_202403', 'CCYC2_202403', 'HBOM8_202403', 'TR937_202403', 'TR956_202403', 'PINS2_202403', 'DVLW4_202403', 'TS040_202403', 'RLAS2_202403', 'RHUS2_202403', 'JNSC2_202403', 'LSHI4_202403', 'TS305_202403', 'TS343_202403', 'LLKS2_202403', 'AGTN1_202403', 'WCAS2_202403', 'RHRS2_202403', 'TS485_202403', 'TS578_202403', 'TS582_202403', 'HITI4_202403', 'KRNK1_202403', 'AENC2_202403', 'LSTC2_202403', 'LPFC2_202403', 'LOOC2_202403', 'TS661_202403', 'TS872_202403', 'TS897_202403', 'RRAC2_202403', 'FKTC2_202403', 'DMTC2_202403', 'WPKS2_202403', 'KSEC2_202403', 'TS946_202403', 'TS954_202403', 'TS964_202403', 'TT065_202403', 'TT081_202403', 'CSPS2_202403', 'SDSS2_202403', 'SHDS2_202403', 'RWES2_202403', 'TT283_202403', 'PMTW4_202403', 'TT364_202403', 'BEYC2_202403', 'TT456_202403', 'MTRN1_202403', 'MKVN1_202403', 'TT561_202403', 'TT562_202403', 'TT563_202403', 'TT564_202403', 'TT565_202403', 'TT566_202403', 'TT567_202403', 'TT568_202403', 'TT574_202403', 'SFRS2_202403', 'TT590_202403', 'TT591_202403', 'TT610_202403', 'TT681_202403', 'TT689_202403', 'TT695_202403', 'TT696_202403', 'C3SKI_202403', 'TT784_202403', 'TT787_202403', 'TT788_202403', 'TT789_202403', 'TT813_202403', 'TT815_202403', 'TT816_202403'])"
187 ]
188 },
189 "execution_count": 16,
190 "metadata": {},
191 "output_type": "execute_result"
192 }
193 ],
194 "source": [
195 "dat = read_pkl(f\"data/{filename}\")\n",
196 "\n",
197 "# Print top level keys, each corresponds to a RAWS site\n",
198 "dat.keys()"
199 ]
200 },
201 {
202 "cell_type": "code",
203 "execution_count": 18,
204 "id": "e0dda99c-077c-4fdd-a5ae-0c3095c2057f",
205 "metadata": {},
206 "outputs": [
207 {
208 "data": {
209 "text/plain": [
210 "dict_keys(['loc', 'RAWS', 'HRRR'])"
211 ]
212 },
213 "execution_count": 18,
214 "metadata": {},
215 "output_type": "execute_result"
216 }
217 ],
218 "source": [
219 "# Check structure within \n",
220 "dat['CPTC2_202403'].keys()"
221 ]
222 },
223 {
224 "cell_type": "code",
225 "execution_count": 17,
226 "id": "e5d9e2e8-90ff-4597-8dd9-4a8f3cec2303",
227 "metadata": {},
228 "outputs": [
229 {
230 "name": "stdout",
231 "output_type": "stream",
232 "text": [
233 "loc\n",
234 " STID : CPTC2\n",
235 " lat : 38.45964\n",
236 " lon : -109.04731\n",
237 " elev : 8124\n",
238 " pixel_x : 565.3953218828111\n",
239 " pixel_y : 509.89701435338947\n",
240 "RAWS\n",
241 " temp: NumPy array of shape (2206,), min: 265.37199999999996, max: 298.15\n",
242 " fm: NumPy array of shape (2206,), min: nan, max: nan\n",
243 " precip_accum: NumPy array of shape (2206,), min: 234.442, max: 329.692\n",
244 " rh: NumPy array of shape (2206,), min: nan, max: nan\n",
245 " solar: NumPy array of shape (2206,), min: nan, max: nan\n",
246 " wind: NumPy array of shape (2206,), min: 0.0, max: 9.836\n",
247 " time_raws: NumPy array of shape (2206,), type object\n",
248 " hours : 2206\n",
249 " rain: NumPy array of shape (2206,), min: nan, max: nan\n",
250 " time: NumPy array of shape (2208,), type object\n",
251 " Ed: NumPy array of shape (2206,), min: nan, max: nan\n",
252 " Ew: NumPy array of shape (2206,), min: nan, max: nan\n",
253 "HRRR\n",
254 " time: NumPy array of shape (2208,), type object\n",
255 " f00\n",
256 " temp: NumPy array of shape (2208,), min: 266.31361616233835, max: 297.0688328224595\n",
257 " rh: NumPy array of shape (2208,), min: 7.522713670093307, max: 100.0\n",
258 " wind: NumPy array of shape (2208,), min: 0.3045713911157861, max: 15.09099812055511\n",
259 " rain: NumPy array of shape (2208,), min: 0.0, max: 0.0\n",
260 " precip_accum: NumPy array of shape (2208,), min: 0.0, max: 0.0\n",
261 " solar: NumPy array of shape (2208,), min: 0.0, max: 1081.7302004975074\n",
262 " soilm: NumPy array of shape (2208,), min: 0.056126406019675175, max: 0.24291703082319627\n",
263 " canopyw: NumPy array of shape (2208,), min: 0.0, max: 0.22516238164522542\n",
264 " groundflux: NumPy array of shape (2208,), min: -233.81325316643472, max: 413.492723688566\n",
265 " Ed: NumPy array of shape (2208,), min: 3.932314579339184, max: 35.68315971965122\n",
266 " Ew: NumPy array of shape (2208,), min: 3.054240651963201, max: 33.45629656041178\n",
267 " descr : Source: HRRR data from 3d pressure model, linear grid interpolated to RAWS location\n",
268 " f01\n",
269 " temp: NumPy array of shape (2208,), min: 266.23398017958317, max: 296.44382028187067\n",
270 " rh: NumPy array of shape (2208,), min: 7.924628703663794, max: 99.89361880234587\n",
271 " wind: NumPy array of shape (2208,), min: 0.857171459405575, max: 17.08500761588227\n",
272 " rain: NumPy array of shape (2208,), min: 0.0, max: 0.0011523953973755687\n",
273 " precip_accum: NumPy array of shape (2208,), min: 0.0, max: 3.818563702589338\n",
274 " solar: NumPy array of shape (2208,), min: 0.0, max: 1085.3123418809882\n",
275 " soilm: NumPy array of shape (2208,), min: 0.05618867931276529, max: 0.24391702788694158\n",
276 " canopyw: NumPy array of shape (2208,), min: 0.0, max: 0.22516238164522542\n",
277 " groundflux: NumPy array of shape (2208,), min: -190.44097420877293, max: 197.81739845084203\n",
278 " Ed: NumPy array of shape (2208,), min: 3.7409142192408273, max: 35.42171589416511\n",
279 " Ew: NumPy array of shape (2208,), min: 2.890177763576445, max: 33.18474475653207\n",
280 " descr : Source: HRRR data from 3d pressure model, linear grid interpolated to RAWS location\n",
281 " f02\n",
282 " temp: NumPy array of shape (2208,), min: 266.3023447171182, max: 296.3819077033915\n",
283 " rh: NumPy array of shape (2208,), min: 8.572388920530683, max: 98.93624709487392\n",
284 " wind: NumPy array of shape (2208,), min: 0.8997371296338974, max: 17.506610622385477\n",
285 " rain: NumPy array of shape (2208,), min: 0.0, max: 0.000808383361969642\n",
286 " precip_accum: NumPy array of shape (2208,), min: 0.0, max: 6.447254505786518\n",
287 " solar: NumPy array of shape (2208,), min: 9999.0, max: 73766.72554235284\n",
288 " soilm: NumPy array of shape (2208,), min: 0.056229391723326284, max: 0.25618360229458087\n",
289 " canopyw: NumPy array of shape (2208,), min: 0.0, max: 0.22516238164522542\n",
290 " groundflux: NumPy array of shape (2208,), min: 0.009999999776482582, max: 1.7789612461724267\n",
291 " Ed: NumPy array of shape (2208,), min: 3.888212152348996, max: 34.55049300780814\n",
292 " Ew: NumPy array of shape (2208,), min: 3.0189887776983633, max: 32.41137694356307\n",
293 " descr : Source: HRRR data from 3d pressure model, linear grid interpolated to RAWS location\n",
294 " f03\n",
295 " temp: NumPy array of shape (2208,), min: 266.49124815086304, max: 296.43641891429814\n",
296 " rh: NumPy array of shape (2208,), min: 8.22678497134034, max: 99.67813077291042\n",
297 " wind: NumPy array of shape (2208,), min: 0.6948617299916177, max: 16.692394681557076\n",
298 " rain: NumPy array of shape (2208,), min: 0.0, max: 0.0019892073336366233\n",
299 " precip_accum: NumPy array of shape (2208,), min: 0.0, max: 6.620534289264193\n",
300 " solar: NumPy array of shape (2208,), min: 9999.0, max: 75118.94727838921\n",
301 " soilm: NumPy array of shape (2208,), min: 0.05633237734665757, max: 0.24704795136399813\n",
302 " canopyw: NumPy array of shape (2208,), min: 0.0, max: 0.22516238164522542\n",
303 " groundflux: NumPy array of shape (2208,), min: 0.009999999776482582, max: 1.82984322300172\n",
304 " Ed: NumPy array of shape (2208,), min: 3.898405875086503, max: 35.3161585254147\n",
305 " Ew: NumPy array of shape (2208,), min: 3.026358431425586, max: 33.1225109478954\n",
306 " descr : Source: HRRR data from 3d pressure model, linear grid interpolated to RAWS location\n",
307 " f04\n",
308 " temp: NumPy array of shape (2208,), min: 266.60861236441275, max: 296.7403710940174\n",
309 " rh: NumPy array of shape (2208,), min: 8.22397927244252, max: 99.218160938994\n",
310 " wind: NumPy array of shape (2208,), min: 0.8755208489425036, max: 16.013336774541347\n",
311 " rain: NumPy array of shape (2208,), min: 0.0, max: 0.0009086639333167349\n",
312 " precip_accum: NumPy array of shape (2208,), min: 0.0, max: 10.47931470944884\n",
313 " solar: NumPy array of shape (2208,), min: 9999.0, max: 74598.02692116056\n",
314 " soilm: NumPy array of shape (2208,), min: 0.05639465063974769, max: 0.26057258465276445\n",
315 " canopyw: NumPy array of shape (2208,), min: 0.0, max: 0.22516238164522542\n",
316 " groundflux: NumPy array of shape (2208,), min: 0.009999999776482582, max: 1.8045979136401873\n",
317 " Ed: NumPy array of shape (2208,), min: 3.685043643149153, max: 35.04682955772404\n",
318 " Ew: NumPy array of shape (2208,), min: 2.841425105087737, max: 32.86979493206988\n",
319 " descr : Source: HRRR data from 3d pressure model, linear grid interpolated to RAWS location\n",
320 " f05\n",
321 " temp: NumPy array of shape (2208,), min: 266.3648542238918, max: 296.2888829803619\n",
322 " rh: NumPy array of shape (2208,), min: 7.537421962815154, max: 99.85815569767279\n",
323 " wind: NumPy array of shape (2208,), min: 0.8003855152656467, max: 16.913225254587914\n",
324 " rain: NumPy array of shape (2208,), min: 0.0, max: 0.0018840534121556014\n",
325 " precip_accum: NumPy array of shape (2208,), min: 0.0, max: 11.387790077406123\n",
326 " solar: NumPy array of shape (2208,), min: 9999.0, max: 64186.98061084289\n",
327 " soilm: NumPy array of shape (2208,), min: 0.05678997317203569, max: 0.27401656887314185\n",
328 " canopyw: NumPy array of shape (2208,), min: 0.0, max: 0.22516238164522542\n",
329 " groundflux: NumPy array of shape (2208,), min: 0.009999999776482582, max: 1.7929638514819835\n",
330 " Ed: NumPy array of shape (2208,), min: 4.116434318104, max: 35.40442852734956\n",
331 " Ew: NumPy array of shape (2208,), min: 3.222569365074795, max: 33.17071185831675\n",
332 " descr : Source: HRRR data from 3d pressure model, linear grid interpolated to RAWS location\n"
333 ]
334 }
335 ],
336 "source": [
337 "print_dict_summary(dat['CPTC2_202403'])"
338 ]
339 },
340 {
341 "cell_type": "code",
342 "execution_count": 19,
343 "id": "f9b9dafc-1020-4973-84e4-321a903441b1",
344 "metadata": {},
345 "outputs": [
346 {
347 "data": {
348 "text/html": [
349 "<div>\n",
350 "<style scoped>\n",
351 " .dataframe tbody tr th:only-of-type {\n",
352 " vertical-align: middle;\n",
353 " }\n",
354 "\n",
355 " .dataframe tbody tr th {\n",
356 " vertical-align: top;\n",
357 " }\n",
358 "\n",
359 " .dataframe thead th {\n",
360 " text-align: right;\n",
361 " }\n",
362 "</style>\n",
363 "<table border=\"1\" class=\"dataframe\">\n",
364 " <thead>\n",
365 " <tr style=\"text-align: right;\">\n",
366 " <th></th>\n",
367 " <th>Band</th>\n",
368 " <th>hrrr_name</th>\n",
369 " <th>dict_name</th>\n",
370 " <th>descr</th>\n",
371 " </tr>\n",
372 " </thead>\n",
373 " <tbody>\n",
374 " <tr>\n",
375 " <th>0</th>\n",
376 " <td>616</td>\n",
377 " <td>TMP</td>\n",
378 " <td>temp</td>\n",
379 " <td>2m Temperature [K]</td>\n",
380 " </tr>\n",
381 " <tr>\n",
382 " <th>1</th>\n",
383 " <td>620</td>\n",
384 " <td>RH</td>\n",
385 " <td>rh</td>\n",
386 " <td>2m Relative Humidity [%]</td>\n",
387 " </tr>\n",
388 " <tr>\n",
389 " <th>2</th>\n",
390 " <td>624</td>\n",
391 " <td>WIND</td>\n",
392 " <td>wind</td>\n",
393 " <td>10m Wind Speed [m/s]surface Precip. Rate [kg/m...</td>\n",
394 " </tr>\n",
395 " <tr>\n",
396 " <th>3</th>\n",
397 " <td>628</td>\n",
398 " <td>PRATE</td>\n",
399 " <td>rain</td>\n",
400 " <td>surface Total Precipitation [kg/m^2]</td>\n",
401 " </tr>\n",
402 " <tr>\n",
403 " <th>4</th>\n",
404 " <td>629</td>\n",
405 " <td>APCP</td>\n",
406 " <td>precip_accum</td>\n",
407 " <td>surface Downward Short-Wave Radiation Flux [W/...</td>\n",
408 " </tr>\n",
409 " <tr>\n",
410 " <th>5</th>\n",
411 " <td>661</td>\n",
412 " <td>DSWRF</td>\n",
413 " <td>solar</td>\n",
414 " <td>surface Total Precipitation [kg/m^2]</td>\n",
415 " </tr>\n",
416 " <tr>\n",
417 " <th>6</th>\n",
418 " <td>561</td>\n",
419 " <td>SOILW</td>\n",
420 " <td>soilm</td>\n",
421 " <td>0.0m below ground Volumetric Soil Moisture Con...</td>\n",
422 " </tr>\n",
423 " <tr>\n",
424 " <th>7</th>\n",
425 " <td>612</td>\n",
426 " <td>CNWAT</td>\n",
427 " <td>canopyw</td>\n",
428 " <td>Plant Canopy Surface Water [kg/m^2]</td>\n",
429 " </tr>\n",
430 " <tr>\n",
431 " <th>8</th>\n",
432 " <td>643</td>\n",
433 " <td>GFLUX</td>\n",
434 " <td>groundflux</td>\n",
435 " <td>surface Ground Heat Flux [W/m^2]</td>\n",
436 " </tr>\n",
437 " </tbody>\n",
438 "</table>\n",
439 "</div>"
440 ],
441 "text/plain": [
442 " Band hrrr_name dict_name \\\n",
443 "0 616 TMP temp \n",
444 "1 620 RH rh \n",
445 "2 624 WIND wind \n",
446 "3 628 PRATE rain \n",
447 "4 629 APCP precip_accum \n",
448 "5 661 DSWRF solar \n",
449 "6 561 SOILW soilm \n",
450 "7 612 CNWAT canopyw \n",
451 "8 643 GFLUX groundflux \n",
452 "\n",
453 " descr \n",
454 "0 2m Temperature [K] \n",
455 "1 2m Relative Humidity [%] \n",
456 "2 10m Wind Speed [m/s]surface Precip. Rate [kg/m... \n",
457 "3 surface Total Precipitation [kg/m^2] \n",
458 "4 surface Downward Short-Wave Radiation Flux [W/... \n",
459 "5 surface Total Precipitation [kg/m^2] \n",
460 "6 0.0m below ground Volumetric Soil Moisture Con... \n",
461 "7 Plant Canopy Surface Water [kg/m^2] \n",
462 "8 surface Ground Heat Flux [W/m^2] "
463 ]
464 },
465 "execution_count": 19,
466 "metadata": {},
467 "output_type": "execute_result"
468 }
469 ],
470 "source": [
471 "# Print dataframe used to organize HRRR band retrievals\n",
472 "band_df_hrrr = pd.DataFrame({\n",
473 " 'Band': [616, 620, 624, 628, 629, 661, 561, 612, 643],\n",
474 " 'hrrr_name': ['TMP', 'RH', \"WIND\", 'PRATE', 'APCP',\n",
475 " 'DSWRF', 'SOILW', 'CNWAT', 'GFLUX'],\n",
476 " 'dict_name': [\"temp\", \"rh\", \"wind\", \"rain\", \"precip_accum\",\n",
477 " \"solar\", \"soilm\", \"canopyw\", \"groundflux\"],\n",
478 " 'descr': ['2m Temperature [K]', \n",
479 " '2m Relative Humidity [%]', \n",
480 " '10m Wind Speed [m/s]'\n",
481 " 'surface Precip. Rate [kg/m^2/s]',\n",
482 " 'surface Total Precipitation [kg/m^2]',\n",
483 " 'surface Downward Short-Wave Radiation Flux [W/m^2]',\n",
484 " 'surface Total Precipitation [kg/m^2]',\n",
485 " '0.0m below ground Volumetric Soil Moisture Content [Fraction]',\n",
486 " 'Plant Canopy Surface Water [kg/m^2]',\n",
487 " 'surface Ground Heat Flux [W/m^2]']\n",
488 "})\n",
489 "\n",
490 "band_df_hrrr"
491 ]
492 },
493 {
494 "cell_type": "markdown",
495 "id": "191d9cb9-bbbd-4a7d-8413-8b508e7be052",
496 "metadata": {},
497 "source": [
498 "## Formatting Data\n",
499 "\n",
500 "The `build_train_dict` function reads the previously described dictionary and processes it in a few ways. The function lives in the `data_funcs` python module, which is intended to include code that is specific to the particular formatting decisions of this project. The `build_train_dict` function can receive some important parameters that control how it processes the data:\n",
501 "\n",
502 "* `params_data`: this is a configuration file. An example is saved internally in this project as `params_data.yaml`. This file includes hyperparameters related to data filtering. These hyperparameters control how suspect data is flagged and filtered.\n",
503 "* `atm_source`: this specifies the subdictionary source for the atmospheric data. Currently this is one of \"HRRR\" or \"RAWS\".\n",
504 "* `forecast_hour`: this specifies which HRRR forecast hour should be used. At the 0th hour, the HRRR weather model is very smooth and there is no accumulated precipitation yet. Within `wrfxpy`, the 3rd forecast hour is used.\n",
505 "\n",
506 "\n",
507 "The `build_train_dict` function performs the following operations:\n",
508 "\n",
509 "* Reads a list of file names\n",
510 "* Extracts FMC and all possible modeling variables. This includes\n",
511 " * Extracting static variables, like elevation, and extending them by the number of timeseries hours to fit a tabular data format for machine learning.\n",
512 " * Calculates derived features like hour of day and day of year.\n",
513 " * Calculates hourly precipitation (mm/hr) from accumulated precipitation.\n",
514 "* Temporally interpolate RAWS data, including FMC, to make it line up in time with the HRRR data. The HRRR data is always on a regular hourly interval, but the RAWS data can have missing data or return values not exactly on the hour requested.\n",
515 "* Shift the atmospheric data by the given `forecast_hour`. So if you want to build a timeseries at 3pm using the 3hr HRRR forecast data, you would start your data with the 3hr forecast from noon.\n",
516 "* Perform a series of data filtering steps:\n",
517 " * If specified, the total timeseries within the input dictioanry is broken up into chunks of a specified number of `hours`. This makes the data filtering much easier, since we want continuous timeseries for training the RNN models, and if chunks of data are missing in time from the RAWS data it is easier to break the whole timeseries into smaller pieces and filter out the bad ones.\n",
518 " * Physically reasonable min and max values for various variables are applied as filters\n",
519 " * Two main parameters control what is fully excluded from the training data:\n",
520 " * `max_intp_time`: this is the maximum number of hours that is allowed for temporal interpolation. Any RAWS site with a longer stretch of missing data will be flagged and removed.\n",
521 " * `zero_lag_threshold`: this is the maximum number of hours where there can be zero change in a variable before it is flagged as a broken sensor and values are set to NaN for that period.\n",
522 " * NOTE: since this is training data for a model where ample input data is available, we will air on the side of aggressively filtering out suspect data since more can always be collected if volume is an issue. It is possible that sensors break nonrandomly, maybe more missing data in a particular season of the year. This merits further study. "
523 ]
524 },
525 {
526 "cell_type": "code",
527 "execution_count": 21,
528 "id": "8a9f18e3-be33-4b29-bcda-6f7c7bb72e6a",
529 "metadata": {},
530 "outputs": [
531 {
532 "data": {
533 "text/plain": [
534 "{'max_intp_time': 10,\n",
535 " 'zero_lag_threshold': 10,\n",
536 " 'hours': 720,\n",
537 " 'min_fm': 1,\n",
538 " 'max_fm': 90,\n",
539 " 'min_rain': 0,\n",
540 " 'max_rain': 100,\n",
541 " 'min_wind': 0,\n",
542 " 'max_wind': 35,\n",
543 " 'min_solar': 0,\n",
544 " 'max_solar': 1400,\n",
545 " 'min_soilm': 0}"
546 ]
547 },
548 "execution_count": 21,
549 "metadata": {},
550 "output_type": "execute_result"
551 }
552 ],
553 "source": [
554 "params_data = read_yml(\"params_data.yaml\") \n",
555 "params_data"
556 ]
557 },
558 {
559 "cell_type": "code",
560 "execution_count": 20,
561 "id": "dc3ea077-2475-414d-a440-fcc3678f1348",
562 "metadata": {},
563 "outputs": [],
564 "source": [
565 "from data_funcs import build_train_dict\n",
566 "\n",
567 "file_paths = f\"data/{filename}\""
568 ]
569 },
570 {
571 "cell_type": "code",
572 "execution_count": null,
573 "id": "61ed37a2-7fe1-464e-8b4f-13dfad131311",
574 "metadata": {},
575 "outputs": [],
576 "source": [
577 "train = build_train_dict(file_paths, atm_source=\"HRRR\", params_data = params_data, spatial=False, verbose=True,\n",
578 " forecast_step = 3)"
579 ]
580 },
581 {
582 "cell_type": "code",
583 "execution_count": null,
584 "id": "d8fb4171-d6d5-484c-890b-cc60ec70de69",
585 "metadata": {},
586 "outputs": [],
587 "source": []
588 },
589 {
590 "cell_type": "code",
591 "execution_count": null,
592 "id": "69b31fc6-f8c6-4d9d-94f8-ebc5e0c8cbb1",
593 "metadata": {},
594 "outputs": [],
595 "source": []
596 },
597 {
598 "cell_type": "code",
599 "execution_count": null,
600 "id": "a1cbfeff-0560-4bbc-beab-801e9dbc8ef5",
601 "metadata": {},
602 "outputs": [],
603 "source": []
604 }
605 ],
606 "metadata": {
607 "kernelspec": {
608 "display_name": "Python 3 (ipykernel)",
609 "language": "python",
610 "name": "python3"
611 },
612 "language_info": {
613 "codemirror_mode": {
614 "name": "ipython",
615 "version": 3
616 },
617 "file_extension": ".py",
618 "mimetype": "text/x-python",
619 "name": "python",
620 "nbconvert_exporter": "python",
621 "pygments_lexer": "ipython3",
622 "version": "3.12.5"
623 }
624 },
625 "nbformat": 4,
626 "nbformat_minor": 5
627 }