SST movies - ENSO, ice

A few days ago, I posted about the movie version of the WebGL presentation of the high resolution NOAA SST data. Carrick had wondered whether regular movies could be made from it, and at the time I didn't know how to.

I can't do anything there from Javascript, because I have no file i/o. Also, the CPU on which JS runs does not itself know what will appear on the screen, since rendering is done by the GPU; it can only ask, which means a PrintScreen call in Windows. But we need to be able to program that.

I messed around with Java and C++ for a while, but then I found some very neat freeware that does what I need. It's called NirCmd, and comes from Nirsoft. It's the sort of software I like - a 43 Kb executable with no installation but lots of very good documentation. You can download from CNet, but there's no need. It does all sorts of windows things, but the particular example I wanted was a one line command to periodically capture the screen to a PNG file.

So then my task was to setup a movie and run it, downloading each frame with NirCMD. I didn't find any way to send signals, so I had to find ways to match the timing. I ran the movie with 4 secs delay, and NirCMD with 3. This risks getting extra shots, but that isn't a problem. Then I used ImageMagick to crop the files and convert to .jpg. Then JPGVideo to convert to .avi, then FFmpeg to convert to .swf. Along the way, I put a timestamp on the WebGL. It starts top left, but you can Shift_click (click with Shift pressed) to move it where you like.

So I have some samples here. I made five movies, all at the medium 1/2° resolution:

• 2012 ENSO Weekly - a weekly picture of the tropical E Pacific through 2012. It shows an alternation of weak El Nino and La Nina conditions, with a strong La Nina at the end,
  Update - I've added 4-day time step version
• 2011 ENSO Weekly - mainly La Nina
• 2010 ENSO Weekly - again weekly E Pacific. A strong early El Nino plume moves to a La Nina jet as the year progresses. One thing I find fascinating is the classic vortex street pattern.
• 2012 Arctic weekly - a weekly picture of the Arctic region. The region with uniform zero anomaly is guaranteed to be ice, but the converse isn't sure - you'll see an anomaly assigned to regions which would have been classified as ice extent. However, it gives a detailed picture, enriched by the neighboring SST patterns.
• Recent 100/2 days - the most recent 100 days at 2 day intervals. It's a more detailed picture of the development of a La Nina plume.

The movie takes a few seconds to download. Works in Firefox, Chrome, but not yet IE. Update - I see that in Chrome it doesn't respond immediately to a movie choice, but does if you then click on the movie. Update - you can right click to bring up a control menu for stepping (forward) etc.

Choose video 2012 ENSO weekly2012 ENSO 4 day 2011 ENSO weekly 2010 ENSO weekly 2012 Arctic weekly Recent 100/2 days

SST - the 3D movie

Well, maybe not quite 3D, but the movie is painted on a globe that you can rotate while it is running.

This is a follow-up to my post yesterday pointing to a new page with NOAA/NCDC SST data displayed on a WebGL globe that you can rotate and zoom.

I have added a movie facility show you can see the dynamics of processes like ENSO. The process works like this. First you select your start date and resolution, as if you were plotting that date. But instead of New Plot, go to the bottom, next to the new Seq button and enter the Step (interval in days) and Number (of frames in the movie).

Then press Seq. This will download the data for the frames. At 1° resolution, this will show as a passable movie, but at 1/4° it will be painfully slow. But let it run. You'll see a pink window just above counting the frames - it will revert to pale green when finished.

Now the payoff - next to the Cyc button you can enter the cycle time for the real movie. For resolution 1/4° you may find that times less than 1 sec will lead to skipping frames; coarser resolutions should do several frames per second (if you really want). Then press Cyc to run the same sequence that you just downloaded, but at the faster speed.

Long sequences at high res will be a load on memory - I estimate about 3-4 Mb per frame. I've been juggling memory vs speed - I can run very fast, but needing 10 Mb per frame.

You can rotate or zoom while the movie is running. You can rerun (Cyc) as you wish. You can go on to do other plots or make more movies, but then you can't rerun the one you had been watching.

I've found that it works as described in Firefox. In Chrome it works, but doesn't show the frames while loading (following Seq) and the red colors don't show to indicate. The movie is OK though. I'm investigating (probably tomorrow).

Where the exact day in sequence is missing, the movie takes the nearest,

It's great for dynamic processes such as ENSO. 1998 is very impressive, even with monthly steps. I'll load more data for this period overnight. I'd recommend starting with low resolution and sequences of 10 frames or so.

A collection of HiRes SST on WebGL globe

I posted recently about WebGL plot of high resolution AVHRR SST data from NOAA/NCDC. It is 1/4° (about 25 km). I noted that representation of coasts should be improved (it has been). The data is daily, and I showed 21 Dec 2012.

I've now put up a page which I hope to maintain regularly by script. I've uploaded data for many days, going back to at least 1996. I'll continue filling it out. There are selection boxes to let you select the date and resolution, and a button to press to download and display. At top resolution downloading 700 Kb for each plot takes a few seconds.

For the last three years year data is daily, then reverting to weekly and eventually monthly. This will improve.

The globe you will see is a trackball - you can rotate with a mouse and zoom with the right button (vertically). The colors represent anomalies in °C.

Here are some examples of the things that can be seen. Some have noted recently that the seas around Svalbard are fairly ice-free for the time of year. Here is a section from the plot of 7 January 2013, slightly zoomed:



And here is an ENSO study. Here is the cold plume currently in the Equatorial East Pacific, again on 7 Jan 2013:


While here is the long warm El Nino jet from 9 March, 2010:
.

It's interesting to contrast the Gulf of Mexico temperatures too.

Here is a super El Nino plume from 24 February, 1998:
.

High Resolution NOAA SST with WebGL

This is another in the series of experiments with dynamic global presentations with WebGL. It is SST anomaly data (AVHRR) from 21 December 2012. In fact, NOAA High Resolution SST data provided by the NOAA/OAR/ESRL PSD, Boulder, Colorado, USA, from their Web site at https://www.esrl.noaa.gov/psd/. This experiment is more about data handling than science; the million points of a 1/4 deg grid (about 25kmn resolution) are challenging. So is the task of actually getting it on your screen; Firefox works for me, but not IE and it's patchy in Chrome. Sometimes restarting the browser makes the difference. It takes a few seconds to show, so I've put it below the jump.

The plot is of anomalies. NOAA also gives the actual temperatures, but their plot is less informative. Most of the range of colors is used in just expressing the equator to pole variation, so you learn little about what is different currently.

As elsewhere, you can rotate the globe as a trackball, or zoom by dragging the right button vertically. I'll discuss some of the data issues below the plot.

The picture

The scale on the right shows the anomaly units in °C. They can be quite large. There appears to be a La Nina jet in mid-Pacific. The North Atlantic variations are considerable, as is the N Pacific cold spot. Note the persistence of warmth north of Norway, where the ice is low for winter.

Download minimising

There are about a million data points. The grid is regular, so no need to send latitude or longitude, and with 256 colors, the base load is a Mb - significant for a web page download.

However, Javascript does not send floating point, so the naive method is comma-separated text, 4-5 bytes per point.

I generally compact data in strings, using 7 bit ascii. So some coding is needed. I first differenced the data. Colors tend to vary smoothly, so most differences are within a range of 64. These can be sent within a byte, leaving some characters over to signal exceptions. One is land, so a few bytes can be used to describe an often long stretch. The data is being listed in latitude bands. Differences that don't fit within the range are expressed as base 64 numbers. That gets the data down to about 700 Kb.

WebGL minimising

GL has always been extravagant with memory to gain performance. People who advise on WebGL follow this custom. I used for a while the R package RGL. For this task they would use triangles with node coords as 3 floats (32 bit) and colors as 4. Since each point is in six triangles, it entails 42 floats.

That's where I started, but all sorts of things broke down. Firebug (which I rely on) groaned, and overflows happened within WebGL itself.

The best remedy of course would be indexing. The GPU system is parallel, so this is probably bad for performance, but for this kind of thing, we can give up some. WebGL allows it, but the index variables must be one or two byte integers. This seems very odd to me, since the point of indexing is to economise with large objects. People work around it with subdivided objects, and that's an option here, but I would have needed at least sixteen such objects.

So I tried other WebGL facilities. The first was using bytes instead of floats for the rgb in color, and dropping the transparency alpha. That gained a factor of two. Then I used the GL option of triangle strips, which cuts the repetition of nodes. But still each has to be done stated twice.

Coastal issues

I don't use any mapping data. It's not easy, anyway, to beat the 25km resolution of the AVHRR data. But the triangularisation has a grain, and if the coastline goes across it, there is a sawtooth representation. I had developed a neat way of switching the grain, which got rid of this. Unfortunately, that doesn't go well with the use of triangle strips, which tend to enforce the grain. I can get around this with introducing degenerate triangles, but decided to leave that for another day.

December TempLS Global Temp down 0.29°C!! (early est.)

This is a bit early in the month to post, but I see Lucia has noticed an early result from GISS suggesting a very big drop, so I was curious as to TempLS's take.

I think in fact it is too early. The most recent release of GHCN was Jan 5, and in all, land and sea, there were only 3213 stations reporting - see map below. That gives sparse land coverage.

I've recently put a big store of monthly data mapped on a globe. Dec 2012 is included, and the areas responsible can be seen. I'll discuss below.

That said, the TempLS analysis, based on GHCNV3 land temperatures and the ERSST sea temps, also showed a big drop in monthly average of 0.22°C for December, down from 0.51 °C in November.

Below is the graph (lat/lon) of temperature distribution for November. I've also included a count and map of the stations that have reported to this date.

GHCN Stations color map, all years and months, with WebGL

This time last year, I posted a map of average 2011 temperatures, using the then novel HTML 5 canvas shading. Now I could post 2012, but local technology has moved on. With XMLHTTPRequest I can post lots of data - I showed the current century of monthly temps here. And with WebGL, I showed here how shading and speed could be improved, basically with use of the GPU. At the time, I didn't know how to integrate that properly with other Javascript code that I use.

I've been working on all those things. I'll post again about my WebGL learning, but I can make it do what I used to do in JS with HTML 5, and much faster. I've now uploaded all months, annual averages for all years back to 1881, and also decadal averages. You can access them from the panel on the right; details below.

WebGL makes possible Google-Earth style navigation, but I've kept as well the older style, where you can click on a flat map to find a point to centre the projection. It's fast now, and keeps the orientation right.

Data details

Each plot is taken directly from unadjusted station readings in GHCN V3 or from the NOAA's ERSST. The latter is a reconstructed SST, and grid averages are treated as a station at the centre.

The anomalies are calculated as described in this post. They are anomalies relative to the present seasonally expected value. The presence or warming is indicated by negative values in the past - however the color scale is adjusted to the mean of each plot, so this isn't obvious. For annual and decadal averages, I require more than three quarters of readings to have been taken (ie 10 months out of 12, etc). Missing values were interpolated by seasonal averages for each station.

You can use Shift-Click with the mouse on the plot to get individual station details.

How it works

The WebGL interface is described here. You can use the Earth as a trackball, as in Google Earth, or you can click on the navigator map, top right. I prefer the latter because it works instantly, and keeps a conventional orientation. Where you click becomes the centre of the Earth view.

You choose date periods from the selection boxes (decade, year and date). Each time you make a change in any one, the data is downloaded and the picture shown. The "All" at the top means the overall mean - ie decade or year. Below is echoed what you have chosen, plus (in case things go wrong) the file that was fetched.

You can zoom the plot by moving the pointer vertically with either the right button pressed, or left and Ctrl. Up is bigger. The second provision is because the right button annoyingly pops up a window when released, which I haven't found out how to suppress.

Holding the Shift key when clicking brings up numeric (and name) data from the nearest station. You can show dots for the stations with the checkbox under the small map.

If you want to link to a selected plot, a URL appears at bottom right. It currently doesn't reproduce rotation and zoom.

Some observations

As I discussed in the visualizing post, with unadjusted data systematic errors accumulate as you go back in time because of station moves etc. This shows up as sustained spatial variability - persistent apparent cool and hot spots. The decadal averages become dominated by this, and that's a good way of identifying culprits.

In more modern times, though, the thing that impresses me is the spatial consistency of the land data. The SST data is also consistent, but that is partly the effect of the reconstruction. Least consistent is often the USA.

Some wrinkles.

WebGL draws triangles as supplied, and you'll sometimes see the effect of this at the sphere edge. You'll also see that I've drawn the map lines about 100 km above the surface. If they go below the triangle surface, they disappear, so this device avoids that.

I've tried to make sure you can't ask for an unavailable file, but if that fails there will be a XMLHTTPRequest popup. It doesn't stop the program. I've also tried to make options disabled when they don't make sense.

Future

I'm planning to make trend plots available in the same frame, and more ambitiously, to ensure that the click info on stations includes a graph of the annual data, using a subset of the climate plotter.