The necessity of TOBS

Recently there has been more said on USHCN adjustments. My recent post on a Steven Goddard plot posted at WUWT produced a response, which conceded the plot was wrong, but Anthony said, defensively,
" The one thing common to all of it though is that it cools the past, and many people don’t see that as a justifiable or even an honest adjustment."

I've previously written in defence of TOBS, as an adjustment which is not only justifiable but necessary. The information requiring it is staring analysts in the face, and they would be negligent to ignore it. I showed a hourly analysis at Ft Collins, Colo, of the effect of the TOBS bias.

But recently, Steven Mosher reminded that the much-loved (by sceptics) John Daly site had posted a much more comprehensive survey, by Jerry Brennan in November 2005. The analysis is here, and his summary datafile, which I will use, is here (text, 29kb), which also identifies the stations.

There are other extensive results files there too, but unfortunately, it's all a jumble of numbers. AFAIK, no graphics. So I thought I would provide some. Below is a histogram of the effect of changing TOB from 5pm to 9am for each of the 190 stations considered by Jerry, and of subsequent changes to midnight (standard). There is also a table from the original paper by Karl et el, 1986, which showed that over the years in the US, about 30% of stations made such a change to 1986. Many more stations would have changed to effectively midnight reporting when MMTS came in. Ther mean effect of the change is 0.66°C cooling. It is no surprise that USHCN adjustments have the effect of "cooling the past".

In the original Karl et el, 1986 paper, there is the following table showing what changes had been made to observing times:



Vose et al 2005 give this updated plot, showing that the change has gone well towards completion:



Because evening TOB has a warm bias, through double counting very warm afternoons, the change to 9am has a cooling effect. Here is a histogram of the putative effects for the 190 stations of changing from 5pm observing time to 9am. Positive effect indicates (in °C) the bias that has to be subtracted from temperatures before the change.



The standard setting is midnight, which some stations already observed, and which would become the setting after MMTS conversion. So here is the required adjustment (in °C) for changing from 5pm to midnight:



In fact, midnight mostly has a slightly greater cool bias than 9am. This can be seen in the plot of change to midnight from 9am:



These are large changes, which don't of course apply in full to all stations. A small number were already using midnight. The rest will mostly need some kind of TOBS adjustment, which will "cool the past". But on the evidence, there's no choice.

Nonsense plots of USHCN adjustments

On 6 May, Anthony Watts posted at WUWT the following plot, which said it represented the effect of USHCN adjustments over the years. WUWT said:

"Yet as this simple comparison between raw and adjusted USHCN data makes clear…



…adjustments to the temperature record are increasing – dramatically. The present is getting warmer, the past is getting cooler, and it has nothing to do with real temperature data – only adjustments to temperature data. The climate reality our government is living in is little more than a self-serving construct."

Dramatically!

The text gave no source, but linked to this USHCN document, which had no such graph. However, it did say you could click for the source, and that led you to this post from Steven Goddard's site.

The title looks rather official, but he did not give the source. But you can see from comments below that he made it himself. And he doesn't give much detail.

He did in a comment link to two tar files of data, on his site. So I downloaded them. They are standard USHCN files of station monthly averages, raw and initial.

I was surprised at the "dramatic" change, because USHCN had published a fairly well known ver 1 plot:



with much more modest changes, and since it's mostly TOBS, such a different version seemed unlikely.

So I did download the data, and did my own calc. I'll show the R code below the jump. But here is the plot:
Update. Zeke points out in a comment below that the files Steven Goddard supplied were in °C, not °F. Since SG had quoted his results in °F, I assumed that was the form of the data - there was no other indication. I've fixed the plot, sticking with °F.
Update,
A commenter, Peter O'Neill, who checked my code, noted perceptively that I had made a mistake on data reading. I allowed four places for a monthly temp, which is OK between -9 and 99, but for -10 or less loses the sign. My mistake was actually in thinking the data was in deg F, so -10 would be unlikely for a CONUS month average, tho apparently not impossible.

So I've fixed it as he suggested. Mainly, the result is a lot smoother. The format error only hurt badly when final and raw were on opposite sides of -10, which is rare and makes a jumpy result.

I now get a small spike in 2014, as Peter did. It is about 0.1°C. I have figured the reason. The average final-raw has a strong seasonal variation. here are the monthly averages for 2013 in hundredths °C:
12, 17, 14, 1, -7, -9, -10, -14, -14, -8, -2, 3
So the first four months of 2014 gets the positive part.







Pretty different. Note the y axis. I just subtracted raw from initial, for all months where both were available, and did an unweighted average over months and stations. That is what Steven Goddard said he did. Personally, I would at least have done by state, and then perhaps with area weighting, But it probably wouldn't make much difference.

Update: Diagnosis

I suggested here that the problem is that SG subtracted the annual average of the final readings from the average of the raw. Zeke, who has the code, has confirmed. But while the final data generally has no missing data, because of FILNET, the raw data has many missing station/months. So the difference of averages of two different populations is formed, and the data, being absolute temperatures, is very heterogeneous. Spatially, but even worse, seasonally.

The spike is clear evidence of the problem. SG got one here, too, in Illinois. It is inevitable.

In 2014, the final set has 1218 values in each of the first four months. Every station. But for raw, there were, respectively, 891,883,883, and 645 for April. A big preponderance of winter month readings in the average. So the raw is cooler than final for that reason, not adjustments.

Update: Some commenters have kindly contributed their own plots. I've appended these at the end.

Here is my R code. The contents of the tar files are in my directories "raw" and "final".

# Code for averaging USHCN adjustment differences
#Written by Nick Stokes, 9 May 2014
# See post at https://moyhu.blogspot.com.au/

# gather and sort data
f=list.files("raw")
x=c(".raw.tavg",".FLs.52i.tavg")
n=substr(f,1,11)  # station code
iy=1890:2014
#  w will be a big matrix of raw and adjusted, in the original units of .01F
# from 1890-2014
# (yr+12 months,125 years,1218 stations,raw+final)
w=array(NA,c(13,125,1218,2))
for(i in 1:length(n)){
  s=n[i];  # looping over stations
  for(j in 1:2){ # raw, then final
    y=x[j];  # read the file
    b=readLines(paste(c("raw/","final/")[j],s,y,sep=""))
    b=gsub("-9999","  NA ",b)
    h=as.numeric(substr(b,13,16))  # Read year
    # Read months
    for(k in 0:11*9+19)h=rbind(h,as.numeric(substr(b,k,k+3)))
    h=h[,h[1,]>1889]  # delete pre 1890
    kk=match(h[1,],iy) # line up
    w[,kk,i,j]=h # add
  }
  if(i%%100==0)print(Sys.time())  # it takes about 80 sec
}

x=w[2:13,,,2]-w[2:13,,,1]  # final-raw (omit years)
x1=colMeans(x,dims=1,na.rm=T)  # mean over months
x2=rowMeans(x1,na.rm=T)  # mean over stations
x2=round(x2/100,3) # round and set to deg F

# x2 are the annual averages Now do plot
graphics.off()
png("ushcn.png",width=800)
plot(1890:2014,x2,pch=19,xlab="Year", ylab="Final-raw in deg F",main="USHCN V2.5 adjustment discrepancy",col="blue")
lines(1890:2014,x2,col="blue")
text(1920,0,"Graph prepared by Nick Stokes, 9 May 2014")
text(1920,-0.05,"post at https://moyhu.blogspot.com.au/ 9 May")
dev.off()

Plots in comments:
Zeke shows this plot of raw and adjusted, and the difference



Here is Zeke's current version of the difference plot. He's using some gridding, which I think is better, but doesn't make a lot of difference (a bit smoother).



Bruce Schuck (sunshinehours) linked max/min plots as well:

More big errors in monthly GHCN

I looked at the April data, since it's about time to post TempLS. And after my experience last month, I now run a simple check on GHCN QCU, just looking at big deviations from average. And yes, there were some.

In Algeria, Mecheria shows -60.8°C. Normal would be 13 (mountains). There is no entry in the CLIMAT form.
Bechar, Algeria shows 0. That is in the CLIMAT, where all temp entries are 0, an obvious error. Normal would be 20.
Port Hardy is back, this time -18.5°C. CLIMAT shows 7.4, but -18.5 for Clyde River, Nunavat. A persistent pattern. Previous similar errors not corrected. In fact, none of the errors (eg Greenland, Denmark) I noted previously have been corrected.
Update. I found another oddity. Last month Greenland showed very high temps, when the CLIMAT form showed no data. Turned out the high numbers entered for March were from Sept, 2013. In April, GHCN also entered data, when there is no CLIMAT form since February. But it didn't stand out as odd. It turns out that it is January 2014 data, which would normally be too cold, but this winter has been warm there.
In March, Denmark got temperatures from last July (hot). In April, it was from January. This is not CLIMAT, which seem right. I guess at this rate May may have caught up.


Fortunately, I can now fix the few spectacular errors. But if I can do such a simple check, I don't know why GHCN can't. And then there are the deviations of about 15°C; I don't know if they are real or not.
Update: I found  a contact email at GHCN. I hope they fix it.

A Kansas airport comparison

I got involved in a discussion at WUWT about recent unusually early high temperatures in S Kansas, and particularly a reading May 5 of 102°F at Wichita. The WUWT contention was that this and other high temperatures could be ignored because they were measured at airports. But there were other NWS Coop readings not at airports, and I drew attention to one, WWXK1, that reported 101°F, and was on the airport land, but a mile from the Wichita ASOS site, and away from airport activity in grassy land. It also turns out to be a well-equipped NWS site.

At WUWT there were objections to a path, parking not too far away. But still, it's open land around, with a not major road and extensive parkland opposite, and I do tend to think the NWS knows what they are doing. It seems a modern enough site (photo below). So I thought the corroboration of the Airport measure was significant.

Anyway, aside from the immediate issue, I thought this conjunction of sites gave a good opportunity to test in local detail how airport siting might affect temperature measurement. They are only a mile apart on flat grassy land, so if it were not for airport perturbations, one might expect temperatures to be very close.

So I downloaded the GHCN daily unadjusted TMAX and TMIN data. The airport (KICT) code is USW00003928 and the NWS is USC00148847.

My first disappointment was that the NWS record starts in May 2010. If anyone knows where to find more, I'd be glad of a pointer, because the site doesn't look that new. So I can't get a trend difference, but mainly I'm interested in just what kind of day to day differences are observed.

I'll show below plots of those differences. The upshot is that the airport site averages about 0.4°C higher on both TMAX and TMIN. Of course any two apparently similar sites separated by a mile might show such a difference. But it is interesting to see how they track.

Here is a Google Map of the airport site overall:



The official airport ASOS thermometer is in the orange ring, and the NWS site is in the light green. The ASOS site is where the usually quoted Wichita temperatures come from. The airport is on the western edge of the city, with just one recent suburb further west from the km wide park across from the NWS.

Anthony Watts at the WUWT post showed a closeup of the ASOS site:



I have to say it doesn't look too bad to me. But let's see.

Here is a closer view of the NWS site. I think the MMTS instrument is about in the middle of the green ring.



Anthony posted this very good shot



which shows the various instrumentation nicely. you can read criticism of the site here and below.

It turns out that they track each other well, so for a general view of the yearly cycle, I'll show just the ASOS site for 2012-3. I show Oct-Sep so you can see a whole summer and a winter. Months are marked in colors to distinguish.



OK, now here is the plot of differences (ASOS-NWS) for TMAX for each day of the five part years. Light grey lines show 1°C intervals - I've put temperatures on the y axis to quantify the scale, but each year is offset.



Red means the airport ASOS is higher, and you'll see that it is indeed. I've marked large deviations in a transparent color when they overlap the adjacent year. The values are quantised, reflecting readings in whole °F.

There are some large spikes, which I think are some kind of error. For example, on March 12, the ASOS showed 15.0°C, and the NWS site 21.7. Both agree the adjacent days were 21-22°C, and the minima were similar, at about 0°C. So I first thought the ASOS might be wrong, although the NWS max was a repeat of the previous day, which is a little suspicious. To add to the confusion, Tutiempo gives rather different readings, with a dip to 14.4°C on 12th. But Tutiempo also has a similar dip at another Wichita airport (JABARA).So I think on balance the error if any is at NWS. One would expect fewer errors at ASOS. Some other such deviations I looked at also looked like glitches at NWS.

Other than spikes, there isn't much interesting seasonally in the TMAX data. The ASOS site is usually about 1°F warmer; the annual average differences are below, in °C:

2010	2011	2012	2013	2014
0.406	0.402	0.616	0.590	0.528

Here is the corresponding plot of daily minima. There are similar spikes, again I think mainly glitches at NWS. But there is now also some seasonal variation, with the ASOS often colder in winter.



I wondered if the inversion might happen on very cold nights, but I can't find such a pattern

2010	2011	2012	2013	2014
0.480	0.167	0.426	0.498	0.208

Conclusion

The airport ASOS site is consistently warmer than its grassy neighbour, but only by about 0.4°C. With just one case, one can't definitely attribute that to airport effect. The winter reversal of minima is interesting, but aside from that there doesn't seem to be any pattern.

A weekend paradox

There's a post by Willis Eschenbach at WUWT, titled Extreme Times. It notes that with an autocorrelated signal, for any fixed observation period, the max for the period is more likely to be at the ends than in the middle. That's not easily intuitive.

They go on to argue that statements like "the millenium ended with its warmest decade" do not reinforce global warming. That's what millenia do.

The argument he's countering says that there's only about a 1 in 100 chance of that happening by chance. No, says Willis, it's more like 1 in 50. True, but that's still only 1 in 50. It doesn't change much.

Anyway, I thought of a more homely version of the paradox. Counting weeks as starting on Sundays, on what day are weekly temperature maxima most likely to occur?

My argument went thus. It's like with TOBS. Warm days often come in spells. A warm spell midweek will probably yield a max for one week. But a warm spell at the weekend may well make a weekly max on both Sat and Sun. So over a year, say, Sundays will show up more in the statistics. In fact, up to twice as often as mid-week.

Anyway, the argument at WUWT went on, so I checked. I have a file of daily max for Melbourne from this post. It's from May 1855 to Nov 2013. I counted. Results below, with an error corrected:

Update I had made a mistake in transposing matrices, which had the result of somewhat exaggerating the effect. It is still there though. I have posted the Melbourne max data as a 7 col array here. It starts Sun 7 May 1855.

Day of Max	Number
Sunday	1273
Monday	1138
Tuesday	1100
Wednesday	1071
Thursday	1101
Friday	1057
Saturday	1534

This doesn't mean, alas, than Nature gives us specially warm weekends in Melbourne. You'd get the same result for minima. Or, if you start the week on Wednesday:

Day of Max	Number
Wednesday	1306
Thursday	1102
Friday	1067
Saturday	1126
Sunday	1038
Monday	1150
Tuesday	1485

TOBS

This logic lies behind the TOBS adjustment for change of resetting times for min/max thermometers. There you divide into 24 hour periods. The difference is that Nature does make a difference between hours of the day. So if you make a split at 5pm, while it does increase both the occurrences of maxima and minima there, at that time maxima are far more likely to occur and be more counted. That's a warm bias. If you shift to 9am, minima will be favored. In the USHCN, there was a trend to move from 5pm reading to 9am reading of min/max thermometers (which sets start of "day"). That needs to be corrected. And yes, since the bias moved from warm to cold, correcting it increases trends.

March GISS Temp up by 0.25°C

GISS has posted its March estimate for global temperature anomaly, and it has risen from 0.45°C in February to 0,70°C in March. That balances the drop from January to February.

The TempLS comparison is murky this month, because of GHCN errors that I described here, here and here. My original estimate was a rise of 0.27°C, but after correcting a sign reversal at Sarh in Chad, that rose to 0.31°C. Then I found that NORD ADB in Greenland had September data instead of march; removing that brought the estimate back to 0.23°C. Then I found that other Greenland stations were also affected, and removing them all made the final estimate also a rise of 0.25°C. But I had little faith in it.

The comparison maps are below the jump.

Here is the GISS map. It shows no sign of the Chad problem, which they must have fixed. But it does show high values in Greenland.


And here, with the same scale and color scheme, is the earlier TempLS map:


GISS was gererally warmer, more so especially in Asia, but less in W N America. I think TempLS was cool because of an adjustment I made because my new scheme for simple harmonics wasn't getting the anomaly base right. I think I have a proper fix for this now. The TempLS map also had the artificially high Greenland data.

Previous Months

February
January 2014
December
November
October
September
August
July
June
May
April
March
February
January
December 2012
November
October
September
August
July
June
May
April
March
February
January
December 2011
November
October
September
August 2011

More data and plots

More GHCN QC errors

My previous post was on some specific problems in the March GHCN data (monthly TAVG QCU). At this stage that data is newly posted, and while the errors should not have passed QC, they will probably be corrected. I imagine that Greenland, for example, will be sorted once the CLIMAT form comes in.

But I'm finding data points going back, which should have been eliminated by QC. It seems there may be a general failure.

For example, in February, Kwajalein, in the Marshall Islands, had an average of -2.3°C. Historically, it's a pretty steady 28°C at that time. That one was in the CLIMAT form, but should have been picked up. Not so far away, in Truk December 2011, it was 0°C. Again in the CLIMAT form.

Port Hardy, on Vancouver Island, BC, is a very strange case. It seems to have an equable climate. But in February, GHCN gives it -27.5°C. -25.3 in January, -26.4 in December 2013. And other icy spells, like -24.1 in December 2011. This time it isn't the CLIMAT reports, which seem OK. Instead, Port Hardy has been getting the data for Clyde River, in Nunavat, Lat 70°N. This seems to have happened at least 13 times since 2009.

But even more spectacularly, La Paz, in Bolivia, reached 86°C in June 2010, and 83°C in June 2011. The latter seems to have been a reinterpretation of the 53°C in the CLIMAT form. La Paz is normally around 7°C in June. That's not the record, though - Oruro in Bolivia in September 2011 reached 90°C. This and the 2010 La Paz rading were not in the CLIMAT forms. They may have been removed. The issue seems to be decimal points.

The problems are not common, and often relate to errors in the data submitted. But I can't see why they aren't being picked up.