New ISTI Temperature database - stations in Google Maps with GHCN

A few days ago, Peter Thorne of NOAA noted in a comment that a new Initiative ISTI, with NOAA involvement, has released a large new database of surface temperatures. It's on a similar scale to BEST, and I'll do comparisons in due course. But as an initial step, I thought it would be useful to gather a Google maps presentation, as I did for GHCN.

The beta release is here. I used the recommended merge (3 Oct version). The data combines TMIN, TMAX and TAVG (and is big!); I extracted the TAVG. There were 39430 stations in the inventory.

This is a big set for the GM application, so I've divided it into 8 regions with about 5000 each. You can look at them all at once if you like, but it will run very slowly. Selecting one or two regions is much better. There is a little map at the right of the display showing where they are. Because the data takes several seconds to download, I've put the map beneath the fold.

The idea of the map is that it shows stations with tags with information that you can pop up by clicking. But the main use is that you can filter by categories. You can choose ranges of start date (of data), end date, duration and altitude. The mechanics are that you make these selections, select regions, and then press one of the colors (for tag). What you've asked for will appear in that color, additional to what was there before. A special (and useful) color is invisible. The range choices combine with "or" logic, so you get "and" by making what you don't want go away. Because it is "or", you need to suppress the "All" button to make other choices. The buttons toggle. The middle columns with gt and lt signs also toggle.

I've included GHCN stations for comparison. There is a checkbox for each database. To compare you'll probably want to display in one color with one box ticked, and in another with the other box ticked (only one at a time). The green pin is useful here, in case of overlaps.

The map starts out blank, waiting for you to choose a region (or two). It's below the jump:

Usage

The map initially shows no markers. You need to select regions. I'd recommend starting with just one. If you click one of the marker colors, you'll then see a mass of markers in that region. You can filter some out with the invisible button - remember to toggle the "All" button when making selections. You could filter out, say, all stations starting after 1850. The selections are only operative if the left radio button is on.

You can ask for a different selection and a different color. It is the color request that creates actions. This second request doesn't erase markers already showing, though it will change the colors of those that qualify.

Note that choosing a small number of regions helps with performance, but other choices which reduce tags on screen do not help (the tags are there but invisible).

Worked example

I was curious about whether ISTI had more really old data than GHCN. As I found earlier, BEST v1 has a similar number of stations overall, but little new before 1850. So I did this:

1. Set Region 6
   
2. Unset All, set StartYr, and change the textbox to 1800.
   
3. Under Actions, click Yellow.
   
4. Unset ISTI, set GHCN
   
5. Under Actions, click Pin.
   

So I see a lot of yellow tags in Europe, some with green pins, and some green pins on their own. Clicking on the pins brings up info including start dates. Some things I notice:

• In UK, GHCN has Gordon Castle, Greenwich and Manchester. ISTI doesn't, but has a Central England, which may cover the last two.
  
Both have a Trondheim in the right place, with similar dates (ISTI 1761-2012, GHCN 1761-1981). But ISTI has TRONDHEIM_VAERNES, about 2.5° further E, from 1762-2011. Duplicate?
• GHCN has Lund, 1753-1773. ISTI not.
  
• ISTI has two Budapest records, both starting in 1780. GHCN has one.
  ISTI has San Fernando, from 1786, GHCN not.
  
• ISTI has two Prague records (PRAHA-KLEMENTINUM,PRAHA-RUZNY), one starting in 1771, the other in 1775. GHCN has Praha-Ruzyne.
  
• ISTI has a record for Vienna and one for Wien. Both start in 1775.
  
• ISTI has OBERSCHLEISSHEIM and UCCLE (Belg), GHCN not.
  

So ISTI has a few extras in that period in Europe, some of which may be duplicates.

September GISS Temp up 0.03°C

The TempLS analysis, based on GHCNV3 land temperatures and the ERSST sea temps, showed a monthly average of 0.53°C for September, up from 0.49 °C in August. Last month also showed a higher rise with late data. There are more details at the latest temperature data page.

Below is the graph (lat/lon) of temperature distribution for September. I've also included a count and map of the stations that have reported to this date.




This spherical harmonics plot is done with the GISS colors and temperature intervals, and as usual I'll post a comparison when GISS comes out.

And here, from the data page, is the plot of the major indices for the last four months:


Here is the map of 4087 "stations" which contributed to this report.

Fancy graphics and climate data

Over the last two years I've been exploring various ways of using interactivity to make climate data more accessible and attractive. I've been trying to maintain a gallery, but that takes time, and the effort has been lagging.

So I want to write a post which just summarizes the techniques used, to act mainly as a catalogue with pointers.

Here is a table of headings

KML, Google Earth	Spaghetti graphs
Trend plots	Earth projections
HTML 5 and triangle meshes	WebGL
Google Maps	Climate Plotter

KML, Google Earth

The first thing I tried was writing Google Earth files, using their API language KML (of the XML family). This was really just to organise the data provided by the various surface temperature collectors (GHCN, GSOD, CRUTEM, BEST etc. I could show the stations with various colored and sized pins; summary information would appear in balloons on clicking. Then I found an even more useful capability with folders. GE lets you turn the folders on and off. So if I list stations in folders according to their decade of commencement, for example, the user can show a plot with and range of decades he wants.

The downside is that the user needs to download the KML files, or KMZ (a zipped equivalent). They are not excessively big. I have a collection here.

Anyway, the posts on this are here:

Fun with Google Earth, GSOD and KML	More Google Earth station data
Still more KML and Google Earth	GHCN KML visualisation by years
GE visualisation of changes to GHCN stations 1990...	A KMZ file for the BEST stations

A combined KMZ file for BEST, GHCN, GSOD and CRUTE...

Spaghetti graphs

I then looked at different ways of making "spaghetti" graphs (many tangled curves) more readable. I began with just color experiments, then fancy patterns, then a GIF animation (more here and here). But a suggestion from Eli, with advice from TheFordPrefect, got me using Javascript.

The user could, by hovering the mouse, select individual components of the plot in black contrast. Another such plot here included a general R program for generating code from a CSV data file. And this ice extent plot doesn't pick out curves but lets you focus on time ranges.

Trend plots

This series started with a ordinary graphic which showed, in a color triangle, how the total of possible trends from a single time series, over any time interval could be visualised. However, this was then adorned with JS facilities for choosing different datasets,. Then options were added so that the time series would be simultaneously plotted and the interval in question, and its trend, displayed. This would be updated by clicking on the colored triangle, or by controls on the time series graph. Numerical information, including later confidence limits, could be displayed. Here is a list:

A JS gadget for viewing temperature trends.
A picture of statistically significant warming.	Observed SST and model trends
Significant trend differences	Significant trends in Foster/Rahmstorf
Combined GMST trend viewer

Earth projections

I've never been happy with the various projections which attempt to render the whole earth on a page. I have preferred spherical projections, even though it requires several views. But JS creates a Google Earth like possibility - a spherical projection in which you can adjust the viewpoint.

Every month recently I have published a flat projection of a spherical harmonics fit to the temperatures anomalies, so this was a clear candidate. The globe views were typically from the corners of a surrounding cube. Examples are here:

A Javascript worldview for surface temp.

April 2012 temperatures up 0.2°C

HTML 5 and triangle meshes

HTML 5 Canvases are relatively new. They offer huge flexibility for interactive graphing. This was developed in the climate plotter to be described, but the first application was to shading global plots. The canvas lets you prescribe colors at the corners of a triangle and it will then shade continuously between them, or at least between two of them. Getting all three right is harder to manage, unlike with Gouraud shading that does it automatically.

The big merit of this is that I can show temperature anomalies with no model fitting, so that colors will be correct at the stations. With the above caveat, it may be that only two thirds will be exactly right, but that's enough to see what is going on. When that is done, the mesh and stations can be shown, and the user can click to bring up station names and details. Again, a spherical projection is used, and can be user-rotated to an arbitrary view. I provide a navigator map (flat projection) on which you can click to choose the point to focus on. Examples are here:

Nov temps displayed with HTML 5	Cherrypicker's guide to station trends
Visualizing 2011 temperature anomalies

WebGL

This new facility, described in a recent post, makes available the full power of GL, and is potentially an improvement on HTML5 with spherical triangle meshing. However, support from browsers is still patchy, and the JavaScript that implements it is more difficult to modify - GL has its own style that it is perilous to disrupt. Some day it will be the best option.

Google Maps

This is a different style of using Javascript. It creates the facilities of GE with KML, but in a browser window. Google makes its API available, and as in Maps style the globe can be traversed and flags shown of coded colors, with balloons of information in response to clicking etc. Control is by the familiar JS gadgets of radio buttons etc.

Google Maps display of GHCN stations, with Javas...

Climate Plotter

I think the most advanced use of Javascript and HTML 5 is with the Climate Plotter. This interactively draws plots from a store of annual climate data. You can form regressions (of just about anything with anything); you can plot against an image background. You can interactively rescale, show different axes with different units; adjust anomaly basis intervals. The history is here:

Interactive JS climate plotter (update)	Interactive climate plotting news.
Interactive JS plotter Ver 2

Snafu with comments

Probably in conjunction with my introducing the Captcha system, my system imposed moderation. I didn't realise this, and so the queue built up. My apologies here, especially to Girma. I've released all the comments. I've also removed the Captcha for the moment. I'll try to make sure moderation is not re-imposed.

WebGL - dynamic global temperature maps

This is an advance on HTML 5 that I've been using for some previous dynamic temperature plots. Some Web browsers now support WebGL, a Javascript version of openGL for 3D plotting. I've been playing with it for monthly temperature maps - here's September.

What you'll notice is a globe that responds to mouse dragging just like Google Earth - in fact I believe GE, at least initially, used a version of WebGL. The main advantage for presentation is better shading - Gouraud shading rather than the rather kludgy HTML5 canvas version.

A downside is that support for WebGL is quite patchy, and implementation depends also on your graphics card. I've found Chrome is fine; Firefox produces a fragment of picture and then gives an error message, and of course IE is way behind. Some other browsers have the capability but disable it by default. I understand the reason is that it creates a vulnerability to DOS attacks which send dynamic but very slow pictures.

I don't think WebGL will replace my HTML 5 versions for a while. I don't have as much control, so I can't, for example, allow you to click on the picture to bring up local station info.

The other downside is that the files are fairly large, and take a few seconds to download. So I've put them below the jump. There is the WebGL version and a snapshot of the HTML5 version. The color scheme is the same - I haven't figured out yet how to put a bar on the WebGL version. It's a direct plot from the TempLS September station anomalies - exact for each station, and shaded elsewhere.

I've added a technical update describing the methods I used here.




So here is the globe. Give it a spin! The left button rotates, the middle button/wheel enlarges, and the right button changes field of view (which can also enlarge).



Here is a snapshot from the HTML 5 version. You can see that the shading is more ragged.

Technical update

I should say more about how this is done. I use my R program which does the interactive monthly presentations, in conjunction with an excellent R package, rgl, in which Duncan Murdoch has a big role. This enables me to show in an R GUI the spinnable globe as you see it (browser permitting).

Then there is an rgl routine writeWebGF, which generates an HTML program with Javascript. I could use that directly, but it is bulky (about 2.5 Mb, mostly data). The JS is beautifully written, but the data layout is extravagant. I've been able to cut it down to about 500 Kb, with some added Javascript. I show that via an iframe, because I need to run the program when loading is complete. Normally that would be a flag in the body element, but Blogger controls that, hence the iframe.

September TempLS Global Temp up 0.04°C

The TempLS analysis, based on GHCNV3 land temperatures and the ERSST sea temps, showed a monthly average of 0.53°C for September, up from 0.49 °C in August. Last month also showed a higher rise with late data. There are more details at the latest temperature data page.

Below is the graph (lat/lon) of temperature distribution for September. I've also included a count and map of the stations that have reported to this date.




This spherical harmonics plot is done with the GISS colors and temperature intervals, and as usual I'll post a comparison when GISS comes out.

And here, from the data page, is the plot of the major indices for the last four months:


Here is the map of 4087 "stations" which contributed to this report.

Housekeeping

I've been battling with some persistent spammers in comments. I've invoked Captcha word verification for the time being - I hope I can remove it when things quieten down.