A picture of statistically significant warming.

This is the third in the series of plots showing color maps of all possible trends that can be derived from a dataset. The first post was designed to show how noisy short-term trends were, and how you could pick almost any color, representing a trend value, and find a period where it applied. But with some Javascript enhancement, it's also a good way of visualizing trends on a graph.

The scientific damper to choice of trends in a noisy signal is the significance test. So I've adapted the figures to show significance. I'm using the device of transparency - the colors just fade away as significance is lost. There is a small change at 99%, a big drop at 95% and a small further fade at 90%. The small changes are hard to see. The test is whether the trend is significantly different from zero. Colors fade when either the period is short or the estimated trend is in fact close to zero.

The data here are monthly temperature anomalies, so there is correlation, which affects significance. I've used the Quenouille correction for loss of dof. It gives results very close to AR(1) modelling. I'll give details.

I have included two new series - the NOAA land only index and the HADSST2 sea surface temperature. You can choose the series and time intervals by using the radio buttons on the right. I have redesigned the plot to make full use of the screen space. Because it overwrites the sidebars, I'll keep it below the jump.

Update.Sometimes the pictures don't appear - it seems to depend on how you get to the page. I've found that going to the home page and then clicking on the "read more" always seems to work. I'll try to find the reasons.Seems better now,

So here is the plot. On the left, each small colored square represents a temperature, and the legend gives the center of each color range. I'm using more colors to give a more continuous gradation. But again, the colors have a gray band near zero and a brown band at about 1.7 C/century, a roughly typical figure for end 20C. If in doubt, just click a region - the numerical value will appear on the right, and the blue and red balls will jump to mark the endpoints of the trend line.

1999-now 1989-now 1960-now 1901-now

Land and Ocean Hadcrut GISSLO NOAA UAH MSU.RSS TempLS

Land Only BEST GissTs CRUTEM NOAAland Sea Surface HADSST2

You can also control the range from the graph at the right. You can click on the red and blue lines to move the corresponding balls to that position. There are also the nudge controls; the red and blue ones control the corresponding balls. The further from the center you click, the bigger the jumps.

I have added two other nudgers. The reason is that I found it quite informative to make a kind of movie by moving the range along by clicking. The purple nudger, top right, moves both balls keeping the separation constant, so you can see, say, how a 10-year trend varies. The gold one moves them apart, but keeps the mean constant. This lets you see if there is any kind of derivative at a point that makes sense.

I'll discuss in detail the color plot for Hadcrut3 for the period from 1989. The reason is the famous question that was asked of Phil Jones as to whether there had been significant warming since 1995. Presumably someone had worked out that that was about as far back as you could go before warming became significant. Anyway, PJ said no, and of course there was then much chatter of "no warming since 1995". But then later he said that it had become significant, and was criticised for changing his mind.

But you can see from the plot here what is happening. If you follow up the right vertical axis (now), the numbers mark the period of trend. Start 1995 is now nearly 17 years ago, and that is indeed the boundary of significance relative to zero trend. You can follow the white diagonal line to see other recent periods with 16 years of trend. They are all significant, generally with a greater warming trend. So "since 1995" is borderline significant partly because it is a fairly short period, but also because among such periods, the estimated slope, though positive and only a little below the recent average (brown, was small enough to tip the balance.

Conversely, there are periods even on the four year boundary where the estimated trend was different enough from zero to be significant. These tend to associate with unusual years. 2003 and 2005 were warm, so there is a period where short-term warming was significant, while 2008-9 was cold, and significant cooling could be observed. There is also a patch of significant cooling bottom left about the time of Pinatubo.

Calculating significance Just a word on how the significance was calculated. Calculation time is important. There are about 5 million dots on these plots, and each one represents a regression over up to a thousand data points. So you can't just use a standard regression package (unless you are blest with much greater patience than I am). Fortunately you can do simple regressions by just working on a few vectors of cumulated sums, so instead of a sum of many you work with just a few differences of two. But that complicates the programming.

In a simple regression the model is:
T~at+b+ε
and if you have a vector y of T observations, J is a vector of times and I is a vector of 1's, then the least squares solution of y=aJ+bI is needed,
or if X is a nx2 matrix (J,I)
, then
(a,b)^*=(X^*X)^-1X^*y

In fact, S=(X^*X)^-1σ2 is the covariance matrix of (a,b), where σ is the variance of the residuals d=(y-aJ-bI).

σ2 is estimated by s2=(d^*d)/(n-1), so the standard error of the trend a becomes sqrt(S₁₁s2).

That assumes the residuals are independent, ie not correlated. But for monthly anomalies this is not generally true. Effectively the se is larger, and the Quenouille correction is to multiply the se by sqrt((1+ρ)/(1-ρ)) where the correlation ρ = Σ (d_td_t-1)/(n-2)/s2.

The numbers of months (dof) are large enough (min 48) that the t-distribution reduces to normal reduces to normal, so I used the conventional 2-sided se measures (abs(trend)/se>1.96 = 95% etc).