Monday, November 21, 2011

Giss, TempLS and the October GMST

Late this month, so GISS is already out. Both GISS and TempLS recorded an almost identical small rise in the October Land/Ocean anomaly. TempLS was up from 0.424 to 0.486°C, and GISS up from 0.48°C to 0.54°C. The graph is below the jump::

Here is the GISS spatial distribution:

and the TempLS spherical harmonics LS fit:

Past months:
SepGISS Sep 11 - down 0.13°C
AugAugust GISS - Very small rise - TempLS map compari...
July Comparisons of TempLS with reader MP's July 2011 ...

Observed SST and model trends

Bob Tisdale has a post at WUWT comparing sea surface temperature trends predicted by a mean of IPCC models versus HADISST observed trends. He notes that the 17 year (204 month) trends do not agree very well.
Update - I've added an appendix showing how the all-trend plot can be used to understand the arithmetic behind the current drop in 17-year trend.

Tamino pointed out that the model mean that Bob used had far less variability than individual model runs, and could not be expected at all to reproduce the decadal variation of observations.

You can see some of this in the following plot, which includes two other SST measures, HADSST2 and HADSST3 in the mix. These of course are far more interdependent than model runs, but you can already see that the model mean is within the variation of the observations, with the exception of an oscillation between about 1930 and 1960.

I'm interested in this, because I have been writing a series of posts here, here. and here, which try to give a wider view of how calculated trends are part of a larger picture, which can indicate whether the choice is special in some way.

To see decadal trend variation in greater breadth, I made an interactive plot similar to the one described here. It's below the jump.

This color plot shows all the trends that you could have created over the period since 1901, over periods greater than 4 years. The white diagonal lines are tracks of constant trend period, shown on the right axis, and you can estimate the 17-year line and follow it to see how it behaves, as indicated in the plots above. Then you can look at nearby trend periods.

You can click on this plot to move the red and blue balls on the plot on the right. There are also controls on the plot - the red and blue bars, and the "nudgers", marke dwith <<<<>>>>. The purple one of these is particularly useful, because if you have set the interval to 17 years (or 30, or whatever), then it moves the balls preserving this interval. Then you can see how the trends are responding to the various features of the plot. Every time you create a trend, the numerical result is written below the graph.

One thing that stands out is the big peak in 1998. That's the reason for the current decline in trend in HADISST particularly, but also visible in HADSST2. It is a deviation from the models, because they do not hindcast such features. The present drop in trend reflects the fact that the peak is nearly 17 years ago, and is reaching its maximum leverage in the trend. That is more influential than the current observations.

So here is the gadget. You may find that it doesn't work in Internet Explorer, which is usually set to disable Javascript (you can change this). You can select the different time periods and datasets with the radio buttons on the right. There is more information on it here.


Appendix - further explanation of the 204-month trend effects.

Here is one of the still plots from the interactive gadget. It is for HADISST, focussing on effects since 1960. I've emphasised the white line corresponding to 17-year trends. In Bob Tisdale's Fig 3 represents the transect along this line. But we can learn more from the full picture.

It's a mass of colors, but horizontal and vertical bar effects are a feature. These correspond to unusual years, and 1998 is a big one. There is a vertical bar of warming trend, for trend periods which lag 1998. Having a warm year at the "future" end of the trend period augments the trend. But if it is at the other end, it has a negative effect, and you see this with the horizontal bar.

Bob made mention of the current dip in HADISST. With this plot, you can see what is influencing that. It's close to the big horizontal bar from 1998, which lowers the trend. And it's close to the vertical bar from 2008/9, which also lowers the trend. There's a small increment from current cooling. The plot gives you a feel for how these add together. 1998 is dominant. If you move down to the 30-year trend, as Bob showed, there is very little recent dip. The reason is that we've moved away from the 1998 horizontal, but not from the 2008 vertical.

Wednesday, November 16, 2011

A picture of statistically significant warming.

This is the third in the series of plots showing color maps of all possible trends that can be derived from a dataset. The first post was designed to show how noisy short-term trends were, and how you could pick almost any color, representing a trend value, and find a period where it applied. But with some Javascript enhancement, it's also a good way of visualizing trends on a graph.

The scientific damper to choice of trends in a noisy signal is the significance test. So I've adapted the figures to show significance. I'm using the device of transparency - the colors just fade away as significance is lost. There is a small change at 99%, a big drop at 95% and a small further fade at 90%. The small changes are hard to see. The test is whether the trend is significantly different from zero. Colors fade when either the period is short or the estimated trend is in fact close to zero.

The data here are monthly temperature anomalies, so there is correlation, which affects significance. I've used the Quenouille correction for loss of dof. It gives results very close to AR(1) modelling. I'll give details.

I have included two new series - the NOAA land only index and the HADSST2 sea surface temperature. You can choose the series and time intervals by using the radio buttons on the right. I have redesigned the plot to make full use of the screen space. Because it overwrites the sidebars, I'll keep it below the jump.

Update.Sometimes the pictures don't appear - it seems to depend on how you get to the page. I've found that going to the home page and then clicking on the "read more" always seems to work. I'll try to find the reasons.Seems better now,

So here is the plot. On the left, each small colored square represents a temperature, and the legend gives the center of each color range. I'm using more colors to give a more continuous gradation. But again, the colors have a gray band near zero and a brown band at about 1.7 C/century, a roughly typical figure for end 20C. If in doubt, just click a region - the numerical value will appear on the right, and the blue and red balls will jump to mark the endpoints of the trend line.

Land and Ocean
Land Only
Sea Surface

You can also control the range from the graph at the right. You can click on the red and blue lines to move the corresponding balls to that position. There are also the nudge controls; the red and blue ones control the corresponding balls. The further from the center you click, the bigger the jumps.

I have added two other nudgers. The reason is that I found it quite informative to make a kind of movie by moving the range along by clicking. The purple nudger, top right, moves both balls keeping the separation constant, so you can see, say, how a 10-year trend varies. The gold one moves them apart, but keeps the mean constant. This lets you see if there is any kind of derivative at a point that makes sense.

I'll discuss in detail the color plot for Hadcrut3 for the period from 1989. The reason is the famous question that was asked of Phil Jones as to whether there had been significant warming since 1995. Presumably someone had worked out that that was about as far back as you could go before warming became significant. Anyway, PJ said no, and of course there was then much chatter of "no warming since 1995". But then later he said that it had become significant, and was criticised for changing his mind.

But you can see from the plot here what is happening. If you follow up the right vertical axis (now), the numbers mark the period of trend. Start 1995 is now nearly 17 years ago, and that is indeed the boundary of significance relative to zero trend. You can follow the white diagonal line to see other recent periods with 16 years of trend. They are all significant, generally with a greater warming trend. So "since 1995" is borderline significant partly because it is a fairly short period, but also because among such periods, the estimated slope, though positive and only a little below the recent average (brown, was small enough to tip the balance.

Conversely, there are periods even on the four year boundary where the estimated trend was different enough from zero to be significant. These tend to associate with unusual years. 2003 and 2005 were warm, so there is a period where short-term warming was significant, while 2008-9 was cold, and significant cooling could be observed. There is also a patch of significant cooling bottom left about the time of Pinatubo.

Calculating significance Just a word on how the significance was calculated. Calculation time is important. There are about 5 million dots on these plots, and each one represents a regression over up to a thousand data points. So you can't just use a standard regression package (unless you are blest with much greater patience than I am). Fortunately you can do simple regressions by just working on a few vectors of cumulated sums, so instead of a sum of many you work with just a few differences of two. But that complicates the programming.

In a simple regression the model is:
and if you have a vector y of T observations, J is a vector of times and I is a vector of 1's, then the least squares solution of y=aJ+bI is needed,
or if X is a nx2 matrix (J,I)
, then

In fact, S=(X*X)-1σ2 is the covariance matrix of (a,b), where σ is the variance of the residuals d=(y-aJ-bI).

σ2 is estimated by s2=(d*d)/(n-1), so the standard error of the trend a becomes sqrt(S11s2).

That assumes the residuals are independent, ie not correlated. But for monthly anomalies this is not generally true. Effectively the se is larger, and the Quenouille correction is to multiply the se by sqrt((1+ρ)/(1-ρ)) where the correlation ρ = Σ (dtdt-1)/(n-2)/s2.

The numbers of months (dof) are large enough (min 48) that the t-distribution reduces to normal reduces to normal, so I used the conventional 2-sided se measures (abs(trend)/se>1.96 = 95% etc).

Sunday, November 13, 2011

A numbers puzzle.

OK, this has nothing to do with climate. But it's connected with the trends gadget. I made a nudger that looked like this <<<<>>>>. If you click on the center symbols, you nudge by one month. Next out by 2, then four, and in this version, the outermost move by 8.

Well, suppose the sequence of symbols continued, in powers of 2, as far as needed. How far can you be sure of being able to move with, say, 2, 3, 4 or 5 clicks. More precisely, what is the least number that requires n clicks.

For n=2, it's 3. For n=3 it's 11 (8+2+1 or 16-4-1). But 4? 5? Can anyone think of a rule?

I can think of an upper bound for any n. It's less that 4^(n-1). But not a general formula.

Friday, November 11, 2011

A JS gadget for viewing temperature trends.

In my previous post I showed a map of all possible trends calculated over sub-time intervals of a period, with a mechanism for selecting different datasets and periods. Since there seem to be ongoing interest in viewing trends, I thought a useful Javascript gadget could be developed from that.

Now when you select a dataset/time combination, a corresponding time series plot is displayed, with two colored balls representing the start and end of each trend period. The numerical data are displayed on the right. There are several ways of controlling the horizontal position of the balls:
  • You can click on the colored triangle. Each position represents a start/end combination, so the balls will move to the endpoints of the fitted regression line, and the slope and locations will appear on the right. It's a realisation of the color on the plot, and should correspond.
  • You can move the balls by clicking on the red or blue bars on the graph.
  • You can nudge, for fine control, using the <<<<>>>> device of the appropriate color. The outside symbols move the ball of that color by 8 months, the next by 4 and the innermost by 1.
Update: It seems to be working now.
Well, it almost works. Unfortunately, on this blog system you can't test JS in the editor - I have to go straight from the home computer (where it worrks) to live. In fact everything is working except the movement of the balls. In particular, you can click on the color plot and the numerical values will appear on the right. The issue with the balls is that I have to move them by absolute position, and so I have to find out what Blogger containers they are in. We'll get there.
Update. I've posted this at another site here. It seems to work correctly there. It's possible Blogger just isn't going to let me set the balls in motion.


Also on the right you'll see a URL. You can copy this and use it to link to produce directly the configuration you have created. My JS kungfu is not yet up to putting it in the address bar.

You'll notice there is a second set of controls for selecting datasets/times. You can use either that or the one on the top graph. I made the second one so that it was close to the bottom graph.

The blue ball is always the rightmost one. If you try to cross them, they will just switch - you'll still get the plot you were asking for.

The calculation is the same as previously - OLS regression with no allowance for autocorrelation. The black curve in the plots of monthly data is a centred 12-month moving average.

Monday, November 7, 2011

GMST trends - a cherrypicker's guide.

I've developed the diagram of this post into an interactive trend-viewer here.

With the conversation about the new BEST data, there has been an uptick in the posts showing plots from WoodForTrees with trends proving something or other. Here is a recent duel.

So I decided to expand the plots from the previous post to maybe make all this redundant. You'll see a plot of all the popular time intervals, with trends presented in a colorful array. Now all you have to do is spot the color you like, and look up the endpoints on the axes. Or if your disposition is wonkier you can try to spot patterns in the way the trends vary.

Some math details - I've taken each of nine indices and done an OLS regression of the monthly temperature against time for each pair of start and end points an various ranges. No prior smoothing, no annual averaging. There is a little colored square indicating the trend, in °C/century, for each pair. I haven't included periods shorter than four years. I've used rainbow colors, except for two ranges which help to calibrate by eye. These are zero trend, in gray, and 1.7°C, representing a recent average, in brown.

The plot itself is an interactive compendium of 36 images, for the 9 datasets and four time ranges. The times are from 1999 to present, 1989 to present, 1960 to present, and 1901 to present. They are presenting different fragments of the same trend array. I chose 1999 to present a popular contention at the moment that over a decade the trend is zero or less. For the land/sea indices there are certainly ranges for which that is true - I think this plot gives a fuller picture.

To use the interactive aspect, you can click on either of the legends in red to get any combination of index and time. Your current choice will be shown in black. As with all these plots that I do, the images are downloaded when requested, so when you first click there is a short pause - however, when you revisit that image, it's now in cache.

Below the jump, I'll say a bit more about some stationary images.

Update - I see now that {'ve posted that the interactive part doesn't work in IE (this seems to be an issue of security settings). It works in Firefox, and I expect that it will work in Safari and Chrome. I have added at the bottom of the post a table of links to the still images.

Firstly, a few more details about the plots. The faint diagonal lines show the trend period, as enumerated by the axis on the right. I don't go below four years, and part of the purpose of this presentation is to show how noisy short term trends can be. You'll see that the BEST plot stops in March 2010 - I did not include the data-deprived months of April and May, and there is no data for later months.

The data sources for the plots are:
The TempLS data is local, but I can post it if it would be useful

Here is the GISS Land/Ocean plot shown over a 20-year range. This image is not interactive. You can see some horizontal and vertical patterns. The vertical stripe at 2008/9 shows that this was a relatively cold year, and so trends that terminate there are likely to be negative, There was indeed a flurry of trend-watching at that time, and this plot shows why. Remember gray is approx zero trend, so the short-term trends based on 2008 go negative.

Another strong feature is the 1998 hot year, showing as a horizontal stripe. That stripe has lower trends, because the peak sits at the start of the trend period. One would expect the vertical bar below to have markedly positive trends, and it doesn't. I think the reason is that there is another feature - the positive horizontal bar produced by the cold Pinatubo years around 1993. The result is positive, but the pattern by eye goes with the horizontal rather than vertical.

Here's HADCRUT3 over the century. The longer time shows how the noise gathers on the shrt-term diagonal, with more sustained patterns as you get away from that. Right in the middle there is a big zero to negative trend region corresponding to the post-1940 cooling. The warm '30's show out as a vertical bar. And towards the top right, the shorter term recent segments have a strong tendency to be positive.

Well, I could go on, but I think this may help explain how to use the plots. So click away!

Here is a table of links to the still pictures:

1999-Link Link Link Link Link Link Link Link Link
1989-Link Link Link Link Link Link Link Link Link
1960-Link Link Link Link Link Link Link Link Link
1901-Link Link Link Link Link Link Link Link Link

Saturday, November 5, 2011

Stopped warming? Paused?

More in the saga of the BEST data and whether "there is no scientific basis for saying that warming hasn't stopped" (Judith Curry). Or in the latest from Judith
"Has the rate of warming continued unabated, or has there been a pause in the warming?"

Judith has now offered a criterion: "Here I define “pause” to mean a rate of increase of temperature that is less than 0.17 – 0.2 C/decade."
Stopped means below zero. Now you might think that, with the short periods involved, there would be some notion of significance involved. But no:
"Note that the short time scales considered here preclude determination of a statistically significant trend at the 95% confidence level, although lack of statistical signficance does not negate the existence of a pause as defined here."

Well, it occurred to me that if any drop, significant or not, below, say, 0.17C/decade is a pause, then we'd be seeing a lot of them over the years. So I thought I would check that out.

The period of time Judith and others is looking at is about ten years. So lets look at ten year periods in the recent past, and see how many "pauses" show up.

I'll start with the BEST land data. I've plotted each set to end 2009 to avoid the BEST problems with running out of data in early 2010. So here is a plot of decade gradients, plotted against the end of the ten year period. I'm using OLS regression.  I've switched to units of °C/century, where the paused criterion is 1.7, shown with a horizontal line:

So a first surprise - BEST isn't currently showing a pause at all. Nowhere near.  Muller was right - and Tamino noted this. The BEST data shows a strong rise right through this decade.

The dataset that has been nearest to showing a pause in recent years is HADCRUT. (Remember, it is Land/Ocean, BEST is Land). So lets see how it looks:

Quite a lot of pauses - in fact, if that's the criterion, more years showing a pause than not.

For completeness, here is Gistemp, which is somewhat in between:

Ten years is a popular period to examine at the moment. It's long enough to have some plausibility, doesn't tangle with the peak of 1998, and can be manoeuvred to show somewhat reduced warming. But I wondered what might have been the perspective in previous years. How hard would it have then been to show reduced trends? Is the present period really different?

So I made a plot of the various intervals that could have been tried, from six years to twenty, in two-year increments. The following plot shows which time intervals, looking back from each year, would have met the "paused" criterion. Again trying to see if there really is a pause now.

The length of trend period, looking back, is shown on the y-axis. The corresponding color shows whether that choice of trend interval, ending in the x-axis year, would have been declared "paused".

So first BEST again. The surprise is that it is quite hard, in this decade, to meet the paused criterion (slope ≤1.7C/century) with the BEST data. Just one or two 6-8 year periods satisfy it. It gets progressively easier going back, and of course in the 70's looking back, warming of 1.7C/century was rarely attained.

To explain further, if you look at the column above 2009, the bottom blue part says that the previous 6 years shoiwed a trend above 1.7 C/century - not "paused". The next red part says the previous 8 years (2002-9) showed a trend less than 1.7, but for 10,12,14,16,18, and 20 year periods, again, blue - no pause. The columns above other years can be read similarly.

Again Hadcrut shows a greater frequency of pauses:

But only very recently, and there's a lot more "pausing" in the '90s.With the longer periods pre-2000, red doesn't really mean a pause - it's getting back to the period before the current warming really started.

Again Gistemp is in between:

Finally, there is the "stop" criterion, where the gradient dips below zero. How often could such a period have been found in the past. Well, BEST again:

Not since 1995 has it been possible to find a period between 6 and 20 years with negative slope, at least in the (6,8,10,12,14,16,18,20)yrs examined.. And again it gets more common back in the 70's. And again Hadcrut does show a few more recently, and Gistemp is in between:


Two of Judith's criteria for slope without statistical test show no established region where they are met, and show no real tendency, independent of choice of interval, to occur in recent years.

A related post (from over a year ago) concerning trends in this decade is here.