Merry Christmas - and a new gallery

It's time for November temperatures with TempLS. The headline number is a cooling - from 0.48°C to 0.36°C. I'll show the usual comparative plot below.

But what I really want to talk about is a new style of presentation. The last few posts have tracked my learning curve with Javascript; now I'm coupling it with the canvas feature of HTML 5. This means I can do client-side graphics - much more flexible and interactive

What this means for November GHCN temperatures is that I can show the anomalies directly - without cell aggregation or smoothing with orthogonal functions. This is done by a triangular mesh and HTML5's neat linear color gradient function. Every station has it's anomaly directly represented with the correct color.

I found this very interesting, as it shows graphically how correlated the anomalies are. Most of the color picture is actually quite smooth. There are exceptions, and because these can be associated with large triangles, it gives an exaggerated effect there. But for the most part, large scale patterns show through well.

So here's the plot of November average temperature anomalies, in °C. It is of course interactive - you can rearrange it, magnify, show the stations, click to see station detail and numbers etc. Use Ctrl+. Ctrl- to get it the right size for your screen. The mechanics are explained below the jump.

How it works

The flat map at top right is your navigator. If you click a point in that, the sphere will rotate so that point appears in the centre. The buttons below allow modification. Set what you want, and press refresh. You can show stations, and the mesh, and magnify 2×, 4×, or 8× (by setting both). You can click again to unset (and press refresh).

Then you can click in the sphere. At the bottom on the right, the nearest station name and anomaly will appear. You may want to have stations displayed here. You'll see two faint numbers next to "stations". This indicates how much your clicked missed the station by (in pixels). It's not really a test of your mousing, but of my getting the alignment right (fiddly).

I'm continuing to work on the shading. The HTML5 function is good, but it works by RGB linear interpolation. The rainbow scheme that I use isn't linear (linear schemes are monochromatic and boring), so they get out of alignment. This sometimes means that one of the corners of the triangle does not get quite the right color. That can happen when there is a lot of change within a triangle.

Comparative plot of recent months, and the usual spherical harmonics fit.

and the Spherical Harmonics map

Previous Months

October
September
August

More data and plots

November GISS is out - down 0.07°C from October

The other surface indices were in notable unison. TempLS was down by 0.124°C, NOAA by 0.13°C and HADCrut by 0.122°C. Satellites showed less change. Here's the plot:


I had been using the GISS index based on V2 of GHCN, but that was discontinued this month. This caused a few days delay as my script, looking at the V2 file, said there was no change. There doesn't seem to be any great discontinuity caused by the change. Of course I'll be using the V3 version in the future.

Below the jump I'll show the comparison of the GISS and TempLS spatial distributions. This month I did another much more detailed spatial distribution based directly on station readings.



Here is the GISS plot of temperature distribution for November 2011:




And here is the TempLS version, using the GISS base years and levels.colors. I use a standard scale, and this is the first time I have overrun - the white patch over the Caucasus. I was reassured to find that GISS also used it's extreme color there. I did the map fairly this month (about 8th) but it still works fairly well.

Previous Months

October
September
August

More data and plots

Significant trends in Foster/Rahmstorf

Having recently finished a post on trends in unadjusted temperature indices, I see that Tamino has posted his code and results for the indices after adjustment for known exogenous effects.

This is of great interest to the kind of study I was doing. There the variations were assumed random, and assessed for significance on that basis. But of course we don't know they are random - it's just an model in the absence of better information. F&R have used more information, so I want to see what the effect is.

First a review of the methods. Some posts starting here looked at the pattern of temperature trends you could create with all possible start and end points over a period. Then I looked at how allowance for statistical significance changed the picture, and then at how a similar picture could be drawn of upper and lower CI's.

I want to show principally the latter analysis. Here's an example which will be enlarged later:


On the left you see the trends marked in color. The x-axis shows the end year of the trend period; the y-axis shows the start. The faint white lines at 45° show constant trend period, shown on the right axis.

On the right you see, in this case, the lower bound trends. That is the highest trend which allows you to say that the observed trend is significantly greater, at 95% confidence. It gives a cool side check to the trend. You can take a value, look up its color, and say that where you see that color or redder, you know the trend significantly exceeds that value.

Next I'll show the plot for all 5 indices used by Foster/Rahmstorf. This comes with the Javascript gadgetry I developed in previous posts. You can click on any point, and on the right you'll see printed the corresponding period and trends. On the corresponding time series plot, the red and blue balls will jump into position to show the trend. On that plot, there are controls that let you move the balls around to different locations. Below the graph, there are a set of radio button controls which allow you to switch the plot to any of the five datasets (giss,noaa,cru,rss,uah) used by Foster and Rahmstorf.

giss  ncdc cru  rss uah

Now the lower bound plot. It shows that trend which is less than the observed trend, but is the highest of those for which the difference from the observed is significant. In most plots, for the longer periods, the color corresponds to about 1.3°C, so the observed (adjusted) trend is significantly higher than this.

giss ncdc cru rss uah

Now the upper bound plot, converse of the above. It shows that trend which is greater than the observed trend, but is the least of those for which the difference from the observed is significant. In most plots, for the longer periods, the color corresponds to about 1.8°C, so the observed (adjusted) trend is significantly lower than this.

giss ncdc cru rss uah

Significant trend differences

In earlier posts I have shown plots of all possibly linear trends in about a century of surface temperature data from various sources. In this post I modified the plot by fading regions where the trend was not significantly different from zero.

But as commenter Frank noted, this is not the only significant difference that might be of interest. So in this post, I'll show some plots that help to bound the range of signficance.

The first plot shows, for each start and end month, the lowest trend that the observed trend would be significantly less than. That is useful for testing predictions that might be failing on the cool side. For example, if you think that there is a claim that the trend over a period should have been 2°C/century, you can see where the actual trend was significantly (at 95%) below that.

The second plot, below the jump, shows the converse. What is the highest trend that was significantly (95%) below the observed. This is probably mainly of interest in establishing whether a trend was significantly above zero, but you might be interested in other values - if you think a theory significantly under-predicts.

The final plot is of the t-statistic - the trend normalized by its standard error. This lets you look at other degrees of significance - mainly relative to zero, but in conjunction with the other plots, you can work out other trend comparisons too.

I've retained the apparatus whereby you can check each point (by clicking) against its plot, echoing the numerical trend value (and period). I've changed from earlier plots of this kind by allowing trend periods down to 1 year.

Purpose

I should at this stage say that my purpose here is to show how the much invoked arithmetic of trend fitting works out. I'm not saying that it is always a good thing to do, and there are cautions about what significance means.

Saying that a trend is significantly different from a base trend is saying that the on the null hypothesis that the data is formed from the base trend plus random noise the observed result is improbable. Cautions:

• Lack of significance does not mean the base trend is right. It just means it is consistent with this data. Many other possibilities would also be consistent.
• Significance does not mean that any physics, say AGW, is disproved. It just means that there may be something more than random variation plus trend.
• And it may not even mean that. It says the result, on the null hypothesis, is improbable. But improbable things happen. There are about half a million dots on the longest plots. If they were independent, and each had a 5% chance of being in a certain range, then that means 25000 significant dots. Even with correlation, you might still expect something like 5% of the area to show as significant. 

Plots

So here is the first plot, showing which trends the observed trend  would be significantly less than. To use it, pick a trend (color) you want to test from the legend. That and bluer redder colors indicate regions where the trend observed was significantly less than the test (color).  I have reversed the coloring custom of earlier posts, marking zero with dark brown, and 1.7°C/Century with gray. For the associated plot and mode of operation, see this post. But note that you can click anywhere on the plot to show the real trend there (shown in text and also by the time series plot).

1999-now 1989-now 1960-now 1901-now

Land and Ocean Hadcrut GISSLO NOAA UAH MSU.RSS TempLS

Land Only BEST GissTs CRUTEM NOAAland Sea Surface HADSST2

And here is the second plot, showing trends where the observed would be significantly greater. The brown marks the edge of the area where the trend is significantly greater than zero.

1999-now 1989-now 1960-now 1901-now

Land and Ocean Hadcrut GISSLO NOAA UAH MSU.RSS TempLS

Land Only BEST GissTs CRUTEM NOAAland Sea Surface HADSST2

And the third plot, which shows the t-statistic, or ratio of the trend to its standard error. For the number of degrees of freedom here, this is distributed normally, and 1.96, marked in brown, is the level of 95% significance. 1.64 is 90%, and 2.58 is 99%. One observation here is that there is only a small fringe region where a choice of a different test level would alter the result..

1999-now 1989-now 1960-now 1901-now

Land and Ocean Hadcrut GISSLO NOAA UAH MSU.RSS TempLS

Land Only BEST GissTs CRUTEM NOAAland Sea Surface HADSST2

Google Map problem fixed

Well, I'm hoping. I posted a while ago a Google Maps rendition of the GHCN surface temperature network. The Javascript mechanics made it possible to select and display all kinds of subsets, and even show a movie.

I heard some reports that people were having trouble making it display. I'd been struggling a bit to make robust Javascript, but that was getting better. There's a complication that I have to have a valid GM ID, but that seemed OK - users shouldn't need one.

Today I found what I think was the problem. I've recently started using an Amazon bucket, and using a newly acquired Web address www.moyhu.org. But it seems that DNS servers don't always get to the right place using that. So I'm using the full Amazon address.

It seems to work now. It's here.

Nov temps displayed with HTML 5

It's time for November temperatures with TempLS. The headline number is a cooling - from 0.48°C to 0.36°C. I'll show the usual comparative plot below.

But what I really want to talk about is a new style of presentation. The last few posts have tracked my learning curve with Javascript; now I'm coupling it with the canvas feature of HTML 5. This means I can do client-side graphics - much more flexible and interactive

What this means for November GHCN temperatures is that I can show the anomalies directly - without cell aggregation or smoothing with orthogonal functions. This is done by a triangular mesh and HTML5's neat linear color gradient function. Every station has it's anomaly directly represented with the correct color.

I found this very interesting, as it shows graphically how correlated the anomalies are. Most of the color picture is actually quite smooth. There are exceptions, and because these can be associated with large triangles, it gives an exaggerated effect there. But for the most part, large scale patterns show through well.

So here's the plot of November average temperature anomalies, in °C. It is of course interactive - you can rearrange it, magnify, show the stations, click to see station detail and numbers etc. Use Ctrl+. Ctrl- to get it the right size for your screen. The mechanics are explained below the jump.

How it works

The flat map at top right is your navigator. If you click a point in that, the sphere will rotate so that point appears in the centre. The buttons below allow modification. Set what you want, and press refresh. You can show stations, and the mesh, and magnify 2×, 4×, or 8× (by setting both). You can click again to unset (and press refresh).

Then you can click in the sphere. At the bottom on the right, the nearest station name and anomaly will appear. You may want to have stations displayed here. You'll see two faint numbers next to "stations". This indicates how much your clicked missed the station by (in pixels). It's not really a test of your mousing, but of my getting the alignment right (fiddly).

I'm continuing to work on the shading. The HTML5 function is good, but it works by RGB linear interpolation. The rainbow scheme that I use isn't linear (linear schemes are monochromatic and boring), so they get out of alignment. This sometimes means that one of the corners of the triangle does not get quite the right color. That can happen when there is a lot of change within a triangle.

Comparative plot of recent months, and the usual spherical harmonics fit.

and the Spherical Harmonics map

Previous Months

October
September
August

More data and plots

A Javascript Gallery

My last post had a Javascript worldmap of GHCN stations. I think this approach is very promising, and I'm hoping to improve with HTML 5 features soon.

Something new - I have an Amazon S3 bucket which I'm using for backup storage. This caused a hiccup with the last post, when I had not adjusted the permissions properly. I could see it, but probably no-one else could. I hope it works now. Please let me know if there are problems.

Anyway, this post is about a gallery that I've compiled with the various recent Javascript and other interactive graphics collected. It's a page on the blog, which you can also get to through the links on top right. There is a table of contents. I'm hoping to keep adding to it.

Google Maps display of GHCN stations, with Javascript

I have been using KML files with Google Earth for displaying temperature stations so that they can be colored and selected according to inventory information. Steven Mosher got me started on this, and he has again pointed me towards using Google Maps interactively.

This is the next in my Javascript learning process. V3 of the Google maps API is well integrated with JS, and the language gives much more flexible control than KML. So in the display presented here, you can select subsets of the GHCN V3 stations according to inventory and data properties, changing their color or making them disappear. Properties include start, end and duration of data, and urban and airport status.

There is also an animation capability. You can set up a movie to run over a period, and you will see the stations in the chosen category appaer and fade reflecting the availability of data in those years.

You can click on each marker to bring up a balloon with information. Rolling the mouse over shows the name of the station.



Implementation of Javascript is not wholly reliable - IE requires tweaking for permission, and there can be clashes with the Blogger software etc. You may find that looking at the original html from here will work better, and will also give an uncluttered screen. An option is to download the html. Use Ctrl+ and Ctrl- to fit well on the screen.






















































More detailed usage information is below the jump.



You have available all the Google Maps controls. On the right are the controls for this widget, in three sections. You should make selections from the middle section. For the selection to be effective, the left button should be checked. All regular buttons act as toggles, and the label reflects the current state; clicking moves to an opposite state. You can write numbers in the text boxes, and toggle the "<" sign to affect the interpretation. The clear button clears the choices, and All overrides.

When you click an action button, all stations which satisfy any one of the chosen conditions will be re-rendered in that color. You may want to make a note of your choice, because the logic can get tangles after a few choices. Of course, to can just use the All button to restore.

To work the movies, choose a time interval and a pause (time between frames in sec). Then click the movie button, where the label will switch to "Show". Then click an action button; currently selected stations will be rendered in that color in each year for which they have data. The years tick over just above the Movie button. Normally you'll start from no stations visible, and they'll become invisible again after data finishes. Even if they start colored, they will still disappear after data ends.

Here are some suggestions:

• At start, the All button will show. Try just changing the colors with an action button.
• Try coloring by urban status. First All and yellow, then clicked the Mixed button, to show Not M. That means Mixed, the GHCN class between Urban and Rural, will stay yellow. Then, with Urban showing (and All not showing - toggle if it is), click the left radio button beside it, and then click pink. Urban stations will then be pink. Then change Urban to Rural, and click cyan. Now you will have a display showing the three different classes.
• Then click Clear, and then set Duration to less than 70. Click Invisible, and you'll be left with the Urban/Rural coloring for stations with at least 70 years data.
• Finally, a movie. Click Clear, All, Invisible. Then choose some movie years - say 1900-2011, and a pause of 1 sec. Click the Movie button; it changes to Show. Then click a color (Yellow, say) to run the movie. It doesn't matter what order you click buttons prior to the Action.

You can run a movie on, say, airports, but be aware that it is using current classifications, so you'll see 19th century airports.

I'll produce more of these, and maybe one for all stations, but data size (loading time) may be a drag.

Giss, TempLS and the October GMST

Late this month, so GISS is already out. Both GISS and TempLS recorded an almost identical small rise in the October Land/Ocean anomaly. TempLS was up from 0.424 to 0.486°C, and GISS up from 0.48°C to 0.54°C. The graph is below the jump::




Here is the GISS spatial distribution:



and the TempLS spherical harmonics LS fit:



Past months:

Sep	GISS Sep 11 - down 0.13°C
Aug	August GISS - Very small rise - TempLS map compari...
July	Comparisons of TempLS with reader MP's July 2011 ...

Observed SST and model trends

Bob Tisdale has a post at WUWT comparing sea surface temperature trends predicted by a mean of IPCC models versus HADISST observed trends. He notes that the 17 year (204 month) trends do not agree very well.
Update - I've added an appendix showing how the all-trend plot can be used to understand the arithmetic behind the current drop in 17-year trend.

Tamino pointed out that the model mean that Bob used had far less variability than individual model runs, and could not be expected at all to reproduce the decadal variation of observations.

You can see some of this in the following plot, which includes two other SST measures, HADSST2 and HADSST3 in the mix. These of course are far more interdependent than model runs, but you can already see that the model mean is within the variation of the observations, with the exception of an oscillation between about 1930 and 1960.


I'm interested in this, because I have been writing a series of posts here, here. and here, which try to give a wider view of how calculated trends are part of a larger picture, which can indicate whether the choice is special in some way.

To see decadal trend variation in greater breadth, I made an interactive plot similar to the one described here. It's below the jump.



This color plot shows all the trends that you could have created over the period since 1901, over periods greater than 4 years. The white diagonal lines are tracks of constant trend period, shown on the right axis, and you can estimate the 17-year line and follow it to see how it behaves, as indicated in the plots above. Then you can look at nearby trend periods.

You can click on this plot to move the red and blue balls on the plot on the right. There are also controls on the plot - the red and blue bars, and the "nudgers", marke dwith <<<<>>>>. The purple one of these is particularly useful, because if you have set the interval to 17 years (or 30, or whatever), then it moves the balls preserving this interval. Then you can see how the trends are responding to the various features of the plot. Every time you create a trend, the numerical result is written below the graph.

One thing that stands out is the big peak in 1998. That's the reason for the current decline in trend in HADISST particularly, but also visible in HADSST2. It is a deviation from the models, because they do not hindcast such features. The present drop in trend reflects the fact that the peak is nearly 17 years ago, and is reaching its maximum leverage in the trend. That is more influential than the current observations.

So here is the gadget. You may find that it doesn't work in Internet Explorer, which is usually set to disable Javascript (you can change this). You can select the different time periods and datasets with the radio buttons on the right. There is more information on it here.

1999-now 1989-now 1960-now 1901-now

HADSST2 HADISST HADSST3 TOS_MODELS

Appendix - further explanation of the 204-month trend effects.

Here is one of the still plots from the interactive gadget. It is for HADISST, focussing on effects since 1960. I've emphasised the white line corresponding to 17-year trends. In Bob Tisdale's Fig 3 represents the transect along this line. But we can learn more from the full picture.



It's a mass of colors, but horizontal and vertical bar effects are a feature. These correspond to unusual years, and 1998 is a big one. There is a vertical bar of warming trend, for trend periods which lag 1998. Having a warm year at the "future" end of the trend period augments the trend. But if it is at the other end, it has a negative effect, and you see this with the horizontal bar.

Bob made mention of the current dip in HADISST. With this plot, you can see what is influencing that. It's close to the big horizontal bar from 1998, which lowers the trend. And it's close to the vertical bar from 2008/9, which also lowers the trend. There's a small increment from current cooling. The plot gives you a feel for how these add together. 1998 is dominant. If you move down to the 30-year trend, as Bob showed, there is very little recent dip. The reason is that we've moved away from the 1998 horizontal, but not from the 2008 vertical.