Daily reanalysis variation - smoothed maps

I have been posting updated daily NCEP/NCAR reanalysis based indices of global temperature. They fluctuate a lot, but give a reasonable estimate of monthly average. For example, in August 2015, after a period of warmth there has been a sudden dive. I also post daily maps, so it is natural to look at these to see what is causing this. Since weather effects take some days to evolve on a global scale, it is unlikely that the sudden changes are coordinated worldwide. They are likely to be locally identifiable.

The current maps, however, make this difficult because of the detailed patchwork of colors. Color mapping for daily data presents a dilemma, because of the high variation over land compared with sea. I thought I had an optimal solution, where the scale ensures that the 256 colors in my rainbow map occur equally often. From an information viewpoint, this makes the best use of our color discrimination capability. But the area is dominated by sea, where a lot of colors are used to resolve a fairly narrow range. That leaves only a few colors to represent the extremes on land, where the colors lurch from red to blue with little in between. But devoting more colors to the land range makes the oceans bland, and in particular ENSO patterns disappear. They are large relative to monthly average spatial variation, but not wrt daily.

I shifted the color scheme toward better land resolution, compromising with ENSO and ocean. But what I really want to talk about is a new scheme using spherical harmonic (SH) smoothing. This enables you to clear away as much as you like of the detail, and show broad swathes of warm/cool. It doesn't show the extremes of variation, but gives a better idea of how much the various regions contribute to the whole integral.

There is quite a lot of work behind the scenes, which I may describe in future. The SH is worked out in Javascript, so you can choose the level of smoothing. The WebGL plot, similar to that on the data page, has a selection box, where you can choose numbers from 3 to 13. The numbers represent approximately the spatial frequency - ie inverse of resolution. Choosing 3 means the characteristic wavelength is 1/3 the circumference etc. There is also the choice of Full, which is not smoothed, but the data shaded directly.

I'm planning to add this to the main page, which will take some data processing. The current model just shows one day, 27 August, when that cold spell took hold. The plot, with some explanation, is below the fold.

You can select a smoothing style from the selection box, then click the Plot New button below. You'll see that at lower resolution patterns seem to move about a bit. But you get a better idea of the relative magnitudes. As usual, the globe is a trackball, which you can drag about. The orient button will realign so the center line is N-S. The scale is in °C.

Update: I've updated to the latest date, 27th. In fact my earlier dating was wrong - I was showing 24 Aug, not 26th, which makes a difference to the cold spots. Antarctica is the apparent source of the big freeze. European Russia is also cold.

NOAA says July hottest month ever

here but I wish they wouldn't. In fact the anomaly was down from 0.87°C in June to 0.81°C. Oddly, that change is exactly what TempLS mesh now shows, while TempLS grid has the same drop (0.04°C) as GISS. Usually it is the other way around.

OK, so it is a warm year, and this was still the warmest July on record. The NOAA claim that it is the warmest month ever (also Tamino) is based on the annual cycle of absolute temperature, whereby ocean-cominated SH summers are cooler than NH, with less seasonal variation.

Why is this a silly point? The NOAA has a sensible discussion here on the reason for using anomalies in preference to absolute - see point 7. Yet they don't seem to be able to stick to it. They keep lapsing into quoting an annual absolute global temperature, and of course regularly quote a ConUS absolute average.

And it just gets them into trouble, pointlessly. The global absolute is got by adding the anomaly to an annual climatology (14°C) taken from a Phil Jones 1999 paper. But the average anomaly is known rather well, the climatology very poorly in comparison. So the sum is worth far less than the parts. Every now and then, a  sceptic raises the 1997 estimate of 62.45°F (16.92°C) for that year and says - see! the world has cooled over 2°C since. NOAA has been forced to add a feeble disclaimer to the 1997 report. But the sceptics are right (for once) to point this out. It just makes the NOAA look dumb. And of course the troubles caused with the absolute average for ConUS (in clumsy hands) are innumerable.

Back to July - we knew that the global absolute has that seasonal  cycle. It doesn't mean anything in terms of climate change, and isn't news. March had a very high anomaly, July less. But July will always exceed March in absolute.

That's one of the main things about anomaly - it contains the news. The information content about weather and climate change. If I tell you that it was 17°C here yesterday, you won't be impressed. The natural question is - what is it normally? Ie, what is the anomaly? And then you find that it is indeed quite warm for an August day.

You can see this news issue in a temperature map. If you see an absolute temperature map for July 2015, it looks like any other July. Sure, it tells you that Melbourne Fl is warmer than Melbourne Australia, and much other climatological information. But it doesn't tell you much about July 2015. For that you need the anomaly map.

NOAA knows all this. I just wish they would stick to it.

ps In other news, August so far is pretty warm. And Arctic Ice is still melting, with 2015 chasing 2011 for third place. Antarctic ice has entered a freezing pause, which may be linked to ENSO.

USHCN adjustments - a case study

In my last post, I linked to a post at Steve Goddard's, grumbling about my comments at WUWT, where I linked to my earlier post showing a breakdown of total adjustment by states. A commenter picked up on New Hampshire, saying
"I particularly like nick’s new hampshire graph with a whole degree warming suddenly applied a few years ago. Their actual temps were obviously not cooperating."

Well, there is indeed a steep rise at about 1991:



The new graph shows that in 1991-2 there was a total rise of about 1°F made up about equally of a TOBS rise and a non-TOBS part. How could that happen? I investigated.

New Hampshire has just 5 USHCN stations, all operating to end 2013, at least. They are Bethlehem, Durham, First Connecticut Lake, Hanover and Keene. It turns out that the first and last underwent big changes about 1991-2. You can look up the relevant metadata here. Location is under the station-level tab, or location tab (map). TOBS is element level.

Bethlehem is actually a composite. The map below shows what happened. B1 lasted to 1 Jan 1992, and then reappeared as B2 about 4 km NE on 1 Jul 1992. Altitude dropped from 1380 ft to 1180 ft, or about 60 m, which would suggest about 0.7F rise in temp. However, the move also took the station out of town, and the cooling effect of that seems to have dominated.



But at the same time, the time of observation moved from 17.00 (warm bias) to 07.00 (cool). That would require an upward adjustment of order 1-2°F.

The total observed rise in adjustment for Bethlehem was 2.46°F, made up of 1.04 TOBS and 1.42 other.

For Keene, the relocation happened 28 Jan 91, and is recorded as moving 2.9 mi NW. Elevation rose 30 ft, but again, it was a move out of town. And again, the TOB changed from 17.00 to 7.00, requiring a warm adjustment.

The total rise for Keene was 3.34°F, made up of 2°F TOBS, and 1.34F other.

So the combined rise was 5.8°F, which divided by 5 in averaging gives indeed the observed NH rise in adjustment of 1.16°F.

USHCN again - adjustments breakdown

There is another post at WUWT based on Steven Goddard's discredited plots of USHCN adjustments. I dealt with that in detail here. What Goddard does is to calculate the effect of adjustment by taking the difference of one set of stations, adjusted, and a different set (a subset) unadjusted. But of course, the difference includes the climatic differences between the disparate station sets, which are not a result of adjustment. And I show there that that dominates, by simply repeating the calc with long term means replacing the monthly data for the non-overlapping part. The result is very similar, showing that the difference is not due to adjustment, or even weather, but to the different climate mix of the stations.

Prof RG Brown was the promoter of the SG graph at WUWT, but he didn't seem to have much interest in where it came from. I think there is still no link to the Goddard source article (which tells nothing anyway). But I can use one of his analogies to describe why the SG approach is all wrong. RGB postulates a process where the growth of young trees is quantified by measuring total height with a tape measure. So, suppose you measured 5 trees with a tape, and 5 other trees with a ruler. You suspect that the tape may be biased high. So you subtract the mean of the two sets of 5, and say that is the difference due to the tape vs ruler.

But of course it isn't. There is no control to say that the tree heights match. The average heights would be different even if all measured with the same measure. OK, in the USHCN the raw stations were a subset of the adjusted, so a closer analogy would be having all 10 measured with tape, and a subset of 5 with ruler. You still can't quantify the ruler/tape by just differencing means of 10 and 5.

Anyway, SG posted a response here. It is worthless, because he won't let go of the nonsense calc. He disputes my observation that TOBS is a major part by showing two graphs, both with his nonsense component. Yes, of course that is then indeed dominant. Most ridiculously, he says:
"Nick also claims that I am comparing two different sets of stations. This is complete BS. USHCN fabricates missing data for almost half their stations. That is an utterly unacceptable practice."
That is a complete non-sequitur. The stations are different places. Whether he thinks they shouldn't be is irrelevant.

A while ago, to counter some other clumsy mis-calcs of USHCN, I posted a breakdown of USHCN adjustments by state. I'll repeat this below the fold showing separately, as NOAA does in its data, the part due to TOBS, the other adjustment (mainly homogenisation), and the total. TOBS is the biggest, and it is the part with a consistent uptrend, for well-established reasons set out here.

Meanwhile, over a year ago, NOAA rolled out its new nClimDiv system, which really makes all this obsolete.

In these plots, the total adjustment is marked in red. Blue is the part due to TOBS, and green is the remainder. You can click on any state abbreviation below the plot to bring up that state's data. Details of the averaging etc is here. You can see that for the USA in total, while the adjustment reaches about 0.9°F, the maximum difference for the non-TOBS component is less than 0.4°F.







ps. This calc solves (mostly) an earlier puzzle that the present day adjustment is not zero, as would normally be expected. The discrepancy is mostly in the TOBS component. I suspect that the TOBS adjustment has been set to correspond to a midnight reading, as most MMTS is nowadays. So the few remaining min/max thermometers are adjusted accordingly, even to present. Well, it's a theory.

Update - methods

Methods are as for the earlier post. Each point plotted is an average of differences for stations. This means that only stations with raw data are included. State averages are unweighted, but the national average is weighted by state area. The code to produce these plots is similar to that posted earlier, and is here

GISS down by 0.04°C in July

The GISS global anomaly average fell from 0.79°C to 0.75°C in July. This is quite close (as expected) to the decrease (now 0.059°C) in TempLS mesh. It is even closer to the 0.039°C drop in TempLS grid.

It is also the same as the 0.04°C drop in the NCEP/NCAR average. Early in August, I said
"NCEP July was fairly close to April, so those are a reasonable guess for July - ie GISS 0.74°C, NOAA 0.78°C. But I wouldn't be surprised to see them a little higher."
Meanwhile GISS June was adjusted down by 0.01.

The spatial pattern is quite similar to that of TempLS and the NCEP/NCAR based average. Maps below the fold.

Here is GISS for July:



And here is TempLS mesh (spherical harmonics, so smoothed):



And here is NCEP/NCAR:


For the last, the pattern is similar but redder. That mainly reflects the use of GISS data to bridge the anomaly base gap between the range of good NCEP/NCAR data and 1951-80.

TempLS shows July anomaly down by 0.05°C

This balances the rise in June, so it is still warm. Canada and India are still to report, but I am posting today, because the next few days will be busy for me. The detailed report is here. The result is almost exactly in line with the drop reported here in the NCEP/NCAR index. The RSS troposphere index showed a drop of about 0.1°C. So it looks likely that July really was cooler. The main features seem to be, as with the NCEP/NCAR index, warmth around the Mediterranean, and cold in NW Russia and N Europe. Fairly cool in E N America. In time, the NCAR index showed a cool spell early-mid July. Maps below the fold.
Update - with most data now in, the result is much the same.

In other news, Arctic ice has been melting fairly briskly; currently it stands about equal to 2011, but with more ice than 2007 or 2012.

Here is the spherical harmonics plot from the report



For comparison, here is the similar map for the NCEP/NCAR index, to the same 1951-80 base:



It looks a lot redder, mainly because I have used GISS data to put it on a 1951-80 base, which had a generally greater rise since then.  Also the spherical harmonics smooth out extremes. TempLS shows cold in India, where ground stations have not reported. This may be spurious and make a small increase in the average when the land results come in.

Here is the attribution plot. Compared with June, there is a further rise in SST, but weaker land contributions, and continuing cold in Antarctica contributing to the result. This may mean that GISS and NOAA, which have a large SST component, may show as relatively warmer, although TempLS should be equally sensitive to SST.

NCEP cooler in July

The Moyhu NCEP/NCAR index (based on reanalysis) was down in July, 0.164°C from 0.204°C in June. And June was fairly cool. July was close to April. It started and ended fairly warm, with a big cold dip in the middle. But I'm not sure surface indices will follow, as I'll explain.

The native anomaly period I've chosen is 1994-2013, which is a period where the reanalysis seems most reliable. I now, in the left column of the table, and below the month averages, show averages relative to GISS and NOAA bases. This uses those indices to fill the gaps (details here); it helps because it allows for the relative differences between months in one period vs another. Here are those tables:

1951-80 (GISS) base NCEP GISSlo Jul 0.715 NA Jun 0.741 0.8 May 0.813 0.76 Apr 0.698 0.74 Mar 0.851 0.9

1901-2000 (NOAA) base NCEP NOAAlo Jul 0.722 NA Jun 0.758 0.88 May 0.835 0.86 Apr 0.719 0.78 Mar 0.867 0.9

The NCEP index has been generally under-predicting the surface data recently. I analysed its performance in earlier years here, and it tracked surface measures well. Is something different now?

I think the explanation is somewhat similar to the effect noted by the recent paper of Cowtan et al on GCMs and surface indices. Like GCMs, reanalyses return the air temperature of a near surface layer. I use sig995, which is basically 0.995 atmospheric, or an altitude of about 40 m. Surface indices use SST over ocean. This is a proxy for the air temperature, which they would like to use but have sparse measurements. It won't be perfect.

Recently, warm temperatures have been driven by rising SST. I did a detailed breakdown here to show this, but it has been the general trend for the last year or more. This year, Feb/Mar warmth was especially in Asia (land), but that faded and since has been dominated by SST. I think the air-based reanalysis may be lagging, and surface measures could be higher for a while.

As said, NCEP July was fairly close to April, so those are a reasonable guess for July - ie GISS 0.74°C, NOAA 0.78°C. But I wouldn't be surprised to see them a little higher.

Update. You can see the map for July anomalies as a globe here. Here is a map in the GISS style (and to 1951-1980 base) I use for the surface report. A lot of warmth in the E Pacific and Mediterranean region. Cool in N Europe and W Russia. Generally moderate in N America, but warm in the Arctic islands.