Scripts/WeatherPlot

Notes

Harriett Neville and I were discussing the local weather. I said I thought we had sub-zero temperatures in Rochester NY fairly often. She was skeptical. I decided to find some data.

After some rummaging, I found the FEDERAL CLIMATE COMPLEX GLOBAL SURFACE SUMMARY OF DAY DATA site. It offers data sets from 1929 to yesterday for many locations around the world. The key links I found were these:

• A description of the site and data formats.
• ftp://ftp.ncdc.noaa.gov/pub/data/gsod/readme.txt

• A list of reporting location codes.
• ftp://ftp.ncdc.noaa.gov/pub/data/gsod/ish-history.csv

• The data is organized as one file per location per year:
• ftp://ftp.ncdc.noaa.gov/pub/data/gsod/1973/725290-14768-1973.op.gz

I downloaded the files I was interested in, from 1973-2009, to a directory and uncompressed them.

Here is a simple verb to extract the high and low temperatures and plot them. The temperatures are reported in Fahrenheit.

The main data processing is as follows:

1. Read all data files into a boxed list. 1. Cut the text files at LF. 1. Drop the 1st line, the title line, from each file. 1. Flatten the boxes to a list of boxed lines. 1. Remove duplicate blanks from each line. 1. Cut each line at blank. 1. Select the date, high temperature and low temperature elements. 1. Remove the '*' decorator from any data, if present. 1. Convert the all the data to numbers. 1. Unbox the data to a 3xN numeric matrix. 1. Remove any days with missing (9999.9) data. 1. Plot the data with a zero line and X labels for each year.

Finding the data, downloading it and plotting it in J took me about an hour. That's about the same length of time it took for me to create this wiki page.

Usage

Example usage:

   roch73_09''

[{{#file: "plotweather.ijs"}} Download script: plotweather.ijs ]

NB. plot weather data
require 'strings dir misc plot statfns'

roch73_09=: 3 : 0
d=. 'C:\Users\me\Downloads\weather'
f=. 1 dir d,'\725290*.op'
data=.|:>>".&.>&.>(('*'~:])#])&.>&.>(2 17 18{])&.>cut &.>deb&.>;}.&.>LF cut &.>fread&.>f
miss=.(9999.9~:1{data)*.9999.9~:2{data
data=.miss#"1 data
years=.0{"1 (10000 100 100 #: 0{data)
mark=.0,}:>:bx -.years=(}.years),0
xmark=. ~.years
pd 'reset'
pd (}.data),0
pd 'xticpos ',":mark
pd 'xlabel ',":xmark
pd 'key high low zero'
pd 'keypos lbi'
pd 'title Rochester NY Daily High and Low Temperatures 1973-2009'
pd 'isi 1280x 800x'
pd 'save bmp ~temp/rochestertemp.bmp 1280x 800x'
pd 'save png ~temp/rochestertemp.png'
)

gn=: 3 : 0
".@(('*'~:])#])y
)

rmd=: 3 : 0
(*./"1 ]9999.9 ~: y)#y
)

rochmin73_09=: 3 : 0
d=. 'C:\Users\me\Downloads\weather'
f=. 1 dir d,'\725290*.op'
datay=.>(<./) L: 0 rmd L: 0 (>@>) L: 2 gn L: 0 (2 17 18&{@cut@deb) L: 0 }. L: 1 LF cut L: 0 fread L: 0 f
years=.0{"1 ]10000 100 100 #: 0{"1 datay
mark=.0,}:>:bx -.years=(}.years),0
xmark=. ~.years
xa=.i.#datay
dats1=.(3 lsfit xa,:(1{"1 datay)) p. xa
dats2=.(3 lsfit xa,:(2{"1 datay)) p. xa
pd 'reset'
pd (|:(1}."1 datay,.dats1,.dats2)),0
pd 'xticpos ',":mark
pd 'xlabel ',":xmark
pd 'key high low highfit lowfit zero '
pd 'keypos lbi'
pd 'title Rochester NY Yearly Minimum High and Low Temperatures 1973-2009'
pd 'isi 1280x 800x'
pd 'save png ~temp/rochestermintemp.png'
)

roch2005=: 3 : 0
d=. 'C:\Users\me\Downloads\weather'
f=. 1 dir d,'\725290*-2005.op'
data=.|:rmd L: 0 (>@>) L: 2 gn L: 0 (2 17 18{]) L: 1 cut L: 0 deb L: 0 ;}.L: 1 LF cut L: 0 fread L: 0 f
xa=.i.}.$data
dats1=.(5 lsfit xa,:(1{data)) p. xa
dats2=.(5 lsfit xa,:(2{data)) p. xa
years=.0{"1 ]10000 100 100 #: 0{ data
mark=.0,}:>:bx -.years=(}.years),0
xmark=. ~.years
pd 'reset'
pd (1}. data,dats1,:dats2),0
pd 'xticpos ',":mark
pd 'xlabel ',":xmark
pd 'key high low highfit lowfit zero '
pd 'keypos lti'
pd 'title Rochester NY Daily High and Low Temperatures 2005'
pd 'isi 1280x 800x'
pd 'save png ~temp/rochester2005temp.png'
)

See Also

Footnotes

<<FootNote>>

Contributed by David Mitchell

---