| blob | ebe65fdecd0e53f047004401229d4e2c75981db9 |
1 {
2 "cells": [
3 {
4 "cell_type": "markdown",
5 "metadata": {
6 "id": "S_RNHEsfKCu7"
7 },
8 "source": [
9 "Upgrading numpy first. pygrib requires up to date numpy while colab has an old version of numpy, perhaps for compatibility with tensorflow. If the upgrade was needed, the runtime kills itself and restarts, then you need to run the notebook again. "
10 ]
11 },
12 {
13 "cell_type": "code",
14 "execution_count": 1,
15 "metadata": {
16 "executionInfo": {
17 "elapsed": 119,
18 "status": "ok",
19 "timestamp": 1633285232385,
20 "user": {
21 "displayName": "WRF Fire",
22 "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s64",
23 "userId": "09833348162306222387"
24 },
25 "user_tz": 360
26 },
27 "id": "4Vy5r9Ug2iBO"
28 },
29 "outputs": [],
30 "source": [
31 "import numpy as np, os\n",
32 "if not [int(i) for i in np.__version__.split('.')] >= [1,20,1]: # check numpy version\n",
33 " print('Upgrading numpy and stopping RUNTIME! When the notebook completes, please run again.')\n",
34 " ! pip install --upgrade numpy # suggested by Efosa, see also https://github.com/jswhit/pygrib/issues/192\n",
35 " os.kill(os.getpid(), 9) # kill the runtime, need to run again from the beginning"
36 ]
37 },
38 {
39 "cell_type": "markdown",
40 "metadata": {
41 "id": "pY4hPeATK9wZ"
42 },
43 "source": [
44 "Installing dependences:"
45 ]
46 },
47 {
48 "cell_type": "code",
49 "execution_count": 2,
50 "metadata": {
51 "colab": {
52 "base_uri": "https://localhost:8080/"
53 },
54 "executionInfo": {
55 "elapsed": 3061,
56 "status": "ok",
57 "timestamp": 1633285235583,
58 "user": {
59 "displayName": "WRF Fire",
60 "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s64",
61 "userId": "09833348162306222387"
62 },
63 "user_tz": 360
64 },
65 "id": "TwqwYQrBLDck",
66 "outputId": "2eb14e59-9297-4dc1-c328-4d3527297eb6"
67 },
68 "outputs": [
69 {
70 "name": "stdout",
71 "output_type": "stream",
72 "text": [
73 "Requirement already satisfied: pygrib in /Users/jmandel/opt/anaconda3/lib/python3.8/site-packages (2.1.4)\n",
74 "Requirement already satisfied: pyproj in /Users/jmandel/opt/anaconda3/lib/python3.8/site-packages (from pygrib) (3.2.1)\n",
75 "Requirement already satisfied: numpy in /Users/jmandel/opt/anaconda3/lib/python3.8/site-packages (from pygrib) (1.21.2)\n",
76 "Requirement already satisfied: certifi in /Users/jmandel/opt/anaconda3/lib/python3.8/site-packages (from pyproj->pygrib) (2020.6.20)\n",
77 "File ‘grib_file.py’ already there; not retrieving.\n",
78 "\n"
79 ]
80 }
81 ],
82 "source": [
83 "! pip install pygrib \n",
84 "! wget --no-clobber https://raw.githubusercontent.com/openwfm/wrfxpy/master/src/ingest/grib_file.py\n",
85 "from grib_file import GribFile # Martin's utility layer on top of pygrib,from wrfxpy\n",
86 "import numpy as np, os # imported before, just in case\n"
87 ]
88 },
89 {
90 "cell_type": "markdown",
91 "metadata": {
92 "id": "dQ-uJI2sy6I3"
93 },
94 "source": [
95 "Create a function to transfer RTMA file in GRIP2 format from the stash. The function returns zero if the file transfer succeeded. If the file is not available, it returns a nonzero value. Note: if needed, maybe in future add more sophisticated checks, check the return code of wget and if the file size is correct."
96 ]
97 },
98 {
99 "cell_type": "code",
100 "execution_count": 3,
101 "metadata": {
102 "executionInfo": {
103 "elapsed": 3,
104 "status": "ok",
105 "timestamp": 1633285235584,
106 "user": {
107 "displayName": "WRF Fire",
108 "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s64",
109 "userId": "09833348162306222387"
110 },
111 "user_tz": 360
112 },
113 "id": "mxZABVDxt0gd"
114 },
115 "outputs": [],
116 "source": [
117 "import subprocess,os\n",
118 "def load_rtma(path,file):\n",
119 " url='http://math.ucdenver.edu/~jmandel/rtma/' + path\n",
120 " if os.path.exists(file):\n",
121 " print(file + ' already exists, removing')\n",
122 " os.remove(file)\n",
123 " try:\n",
124 " ret = subprocess.check_output(['wget','--no-clobber','--output-document='+ file, url,],stderr=subprocess.STDOUT).decode() # execute command from python strings\n",
125 " if os.path.exists(file):\n",
126 " print('loaded ' + url + ' as ' + file)\n",
127 " return 0\n",
128 " else: \n",
129 " print('file transfer completed, but the file is missing? ' + url) \n",
130 " return 1\n",
131 " except:\n",
132 " print('file transfer failed: ' + url)\n",
133 " return 2\n"
134 ]
135 },
136 {
137 "cell_type": "markdown",
138 "metadata": {
139 "id": "THI6gElyHOOc"
140 },
141 "source": [
142 "Testing the file transfer:"
143 ]
144 },
145 {
146 "cell_type": "code",
147 "execution_count": 4,
148 "metadata": {
149 "colab": {
150 "base_uri": "https://localhost:8080/"
151 },
152 "executionInfo": {
153 "elapsed": 467,
154 "status": "ok",
155 "timestamp": 1633285236049,
156 "user": {
157 "displayName": "WRF Fire",
158 "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s64",
159 "userId": "09833348162306222387"
160 },
161 "user_tz": 360
162 },
163 "id": "Dwbt4UXfro5x",
164 "outputId": "99320267-88f4-4a7a-ba4f-944ccfccde1b"
165 },
166 "outputs": [
167 {
168 "name": "stdout",
169 "output_type": "stream",
170 "text": [
171 "loaded http://math.ucdenver.edu/~jmandel/rtma/20180518/00/temp.grib as temp.grib\n"
172 ]
173 }
174 ],
175 "source": [
176 "year, month, day, hour, variable = (2018, 5, 18, 00, 'temp') # year, month, day, hour, variable\n",
177 "file = variable + '.grib'\n",
178 "path = '%4i%02i%02i/%02i/%s' % (year, month, day, hour, file)\n",
179 "\n",
180 "ret=load_rtma(path,file) # load the file; ret is the return value that can be tested in application\n",
181 "\n"
182 ]
183 },
184 {
185 "cell_type": "markdown",
186 "metadata": {
187 "id": "_yRu_7WvHc6P"
188 },
189 "source": [
190 "Trying to read the file and a basic sanity check if the values make sense:"
191 ]
192 },
193 {
194 "cell_type": "code",
195 "execution_count": 5,
196 "metadata": {
197 "colab": {
198 "base_uri": "https://localhost:8080/"
199 },
200 "executionInfo": {
201 "elapsed": 978,
202 "status": "ok",
203 "timestamp": 1633285237026,
204 "user": {
205 "displayName": "WRF Fire",
206 "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s64",
207 "userId": "09833348162306222387"
208 },
209 "user_tz": 360
210 },
211 "id": "0uSEAB1dZc7P",
212 "outputId": "ad890053-1452-49c2-e955-966973932827"
213 },
214 "outputs": [
215 {
216 "name": "stdout",
217 "output_type": "stream",
218 "text": [
219 "min lats 20.191999000000006 max 52.80766860325035\n",
220 "shape (1377, 2145)\n",
221 "min lons -130.10343806997807 max -60.885557729221034\n",
222 "shape (1377, 2145)\n",
223 "min temp 266.84000000000003 max 315.01\n",
224 "shape (1377, 2145)\n"
225 ]
226 }
227 ],
228 "source": [
229 "gf = GribFile(file)[1] # grib file consists of \"messages\" (basically, variables), rtma files have only one\n",
230 "lats, lons = gf.latlons() # grid of geo coodinates (computed), should be the same for all rtma files here\n",
231 "lats = np.array(lats) # tuple to numpy array\n",
232 "lons = np.array(lons) # tuple to numpy array\n",
233 "temp = np.array(gf.values()) # the actual variable values (here, T at 2m in K)\n",
234 "\n",
235 "print('min lats %s max %s' % (np.amin(lats),np.amax(lats)))\n",
236 "print('shape',lats.shape)\n",
237 "print('min lons %s max %s' % (np.amin(lons),np.amax(lons)))\n",
238 "print('shape',lons.shape)\n",
239 "print('min temp %s max %s' % (np.amin(temp),np.amax(temp)))\n",
240 "print('shape',temp.shape)\n"
241 ]
242 },
243 {
244 "cell_type": "markdown",
245 "metadata": {
246 "id": "-u_piG3uICUA"
247 },
248 "source": [
249 "One special grib file with the terrain height is stored at the root of the stash. This file is a part of the RTMA dataset but no need to download and store every hour, the data should never change. Trying to read it and doing a sanity check. Also,checking if the grid coordinages in this file are the same as before.\n"
250 ]
251 },
252 {
253 "cell_type": "code",
254 "execution_count": 6,
255 "metadata": {
256 "colab": {
257 "base_uri": "https://localhost:8080/"
258 },
259 "executionInfo": {
260 "elapsed": 1452,
261 "status": "ok",
262 "timestamp": 1633285238477,
263 "user": {
264 "displayName": "WRF Fire",
265 "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s64",
266 "userId": "09833348162306222387"
267 },
268 "user_tz": 360
269 },
270 "id": "JZlX8BVl4HRB",
271 "outputId": "77812006-9b98-4936-961c-8672822abfdc"
272 },
273 "outputs": [
274 {
275 "name": "stdout",
276 "output_type": "stream",
277 "text": [
278 "ds.terrainh.bin already exists, removing\n",
279 "loaded http://math.ucdenver.edu/~jmandel/rtma/ds.terrainh.bin as ds.terrainh.bin\n",
280 "min height -76.60000000000001 max 4156.900000000001\n",
281 "shape (1377, 2145)\n",
282 "difference in lats 0.0 lons 0.0\n"
283 ]
284 }
285 ],
286 "source": [
287 "hf='ds.terrainh.bin' # terrain height, same in rtma at all times\n",
288 "load_rtma(hf,hf)\n",
289 "gf = GribFile(hf)[1] \n",
290 "hgt = np.array(gf.values()) # height in m\n",
291 "print('min height %s max %s' % (np.amin(hgt),np.amax(hgt)))\n",
292 "print('shape',hgt.shape)\n",
293 "hlats, hlons = gf.latlons() # grid of geo coodinates (computed), should be the same for all rtma files here\n",
294 "hlats = np.array(hlats) # tuple to numpy array\n",
295 "hlons = np.array(hlons) \n",
296 "print('difference in lats %s lons %s' % (np.amax(np.absolute(lats-hlats)), np.amax(np.absolute(lons-hlons))))\n"
297 ]
298 }
299 ],
300 "metadata": {
301 "colab": {
302 "authorship_tag": "ABX9TyP4ue0fNwFqLrtHTL/TAAi3",
303 "collapsed_sections": [],
304 "name": "load-rtma-demo.ipynb",
305 "provenance": []
306 },
307 "kernelspec": {
308 "display_name": "Python 3",
309 "language": "python",
310 "name": "python3"
311 },
312 "language_info": {
313 "codemirror_mode": {
314 "name": "ipython",
315 "version": 3
316 },
317 "file_extension": ".py",
318 "mimetype": "text/x-python",
319 "name": "python",
320 "nbconvert_exporter": "python",
321 "pygments_lexer": "ipython3",
322 "version": "3.8.5"
323 }
324 },
325 "nbformat": 4,
326 "nbformat_minor": 1
327 }