Monday, October 1, 2012

A necessary adjustment - Time of Observation



Sceptics complain a lot about adjustments made in indexing temperatures. Rarer is an acknowledgement of the argument for the adjustments. The fact is that if an adjustment is appropriate, then it is required. It's not optional.

This post will set out the quantitative basis for one of the larger adjustments to USHCN, a frequent object of this complaint. This is TOBS, the time of observation. It arises because USHCN gets its data from a wide variety of observers, many voluntary. The time at which min/max thermometers are read and reset is recommended but not mandated, but is on record. For many stations it has changed, and this matters.

In this post I take a USCRN station, Boulder, Colorado, with hourly data from 2009-2011. I calculate the effect of varying the notional reading time of a min/max thermometer. There is a positive bias of about 1.3°F if it is read in mid-afternoon, tapering to nearly nil around midnight. There is potentially a cooling bias in the morning, though for this site it was small.

But firstly, a discussion of why temperature measurement is relevant to the climate debate, and what kind of measure should be used.



The role of temperature measurement

The climate debate is about potential global warming caused by the addition of carbon dioxide and other greenhouse gases to the atmosphere. Sometimes the impression is created that the basis for worry is in fact the observation of rising temperatures, and if doubt can be cast on this observation, the worry goes away.

This is not true. The case for AGW is now, and always has been, based on the physics of the greenhouse effect. Addition of GHG's to the atmosphere leads to warming. The amount of warming (climate sensitivity) is not perfectly known, but there are reasonable estimates.

GHG's have increased, so a rise in temperature should be discernible. If there were none, then AGW would be in doubt. That is why a study of historical measurements is important. But history shows that temperatures have risen. There are of course uncertainties about all measurements, and there are short-trerm fluctuations which can obscure trends. But the rise is consistent with AGW. It is not the proof of AGW.


Measuring daily temperature

Every now and then a post like this appears, in which someone discovers that the measure of daily temperature commonly used (Tmax+Tmin)/2 is not exactly what you'd get from integrating the temperature over time. It's not. But so what? They are both just measures, and you can estimate trends with them.

The reason (Tmax+Tmin)/2 is used is that a very long history is available. In pre-electronic days, observers used min-max thermometers like the one on the right. The pins are pushed up as the mercury rises, and do not descend until reset. Note that the minimum scale is reversed. Typically, once a day the position of the pins is read and the pins are moved (eg with magnet) back to sit on the current position of the mercury. The reading gives the max and min of the previous day, but the time when they occurred is not shown. Typically an observer would record the location of the pins and the temperature at the time of reading.

Regular hourly readings are only widely available since the introduction of MMTS a couple of decades ago.

Note that with the minmax thermometer, if you reset the max when the temperature is falling, it may happen that the temperature may not return to that level for the whole next day. In that case, the next max you read will be the value that you set it to. This is, as I'll show, why time of observation matters.

USHCN Adjustments

Discussions of USHCN adjustments often refer to this plot:

Note that is is V1, and so out of date, but it does show the TOBS effect. This paper of Vose et al has more details, and describes the underlying cause thus:
[3] The majority of weather stations in the U.S. Cooperative Observing Network (and therefore in HCN) are staffed by volunteers. Consequently, the network has no mandatory time at which daily measurements must be taken. Most individuals prefer observing times other than midnight, resulting in an observation day that differs from the standard calendar day. For example, at a station where the volunteer reads the thermometers at 0800 LST, the observation day extends from 0800 LST the previous day to 0800 LST on the current day. At a station where the volunteer reads the thermometers at 1700 LST, the observation day starts and ends 9 hours later. Nevertheless, the observations at both stations are recorded for the same calendar day.

[4] When the observation day differs from the calendar day, a ‘‘carry over’’ bias of up to 2.0°C is introduced into monthly mean temperatures. This bias occurs when atmospheric conditions cause a temperature from one day to be ascribed to the following day. For instance, suppose an observer reads the maximum and minimum thermometers at 1700 LST on April 1, then a cold front passes through the area overnight. If the temperature on April 2 never exceeds the value at 1700 LST on April 1 (when the thermometers were last reset), then the recorded maximum will actually be the temperature at 1700 LST on April 1. This temperature will be higher than if the 24-hour measurement ended at midnight, and because the monthly mean is computed by averaging the daily maximums and minimums, the mean for April will likewise be artificially high. In general, this carryover phenomenon results in a warm bias for observation days ending in the afternoon and a cool bias for those ending in the morning.

Station observations - Boulder Colorado

As mentioned, I was looking around for a station with a good set of hourly readings for some years, with few missing values. I first looked at Washington, DC, but there were lots of gaps. So I thought the USCRN station at Boulder was promising, and indeed it had 2009-2011 with only 38 missing hours (which I interpolated). To simplify, I used MST (Mountain Standard Time) only, no daylight saving. All temperatures are in Fahrenheit.

Diurnal pattern

The diurnal pattern varies through the year. But here is a graph of the hourly averages (°F) for all of those three years. As expected there is an afternoon maximum and a minimum in the early morning.

Time of observation effect

This is simulated, supposing that we took the max and min of 24-hour blocks. Often the max measured is the afternoon max of each day, and the min is the early morning min. Then the time of observation doesn't matter.

But sometimes the reset value is not reached again in the next 24 hours. Then the "max" recorded is not a real max - it reflects the warmth of the previous 24-hr period, rather than the cold of the next. So it is a warm bias. If the same thing happens with the min, it is a cold bias.

The following tableau shows the frequency of times of max measurement subject to this notional reset. Here and later it is assumed that the reset occurred just before the time stated. The times are 0:00, 6:00,9:00,14:00,17:00,21:00 MST. Any of the plots can be expanded - just right click and View.



If you look at the first plot, with reset at midnight, you see the expected afternoon peak of maxima, but also a peak at midnight. This says that about 80 times in 3 years the maximum of a calendar day occurred at midnight. This implies that the weather turned cold after midnight. The max doesn't reflect how cold it became. There is a smaller peak at 11pm, reflecting the fewer occasions on which a warm front came through.

Resetting at 6 am, there are still a few days where there is a measured max there. But moving on to 5 pm, there is now a very marked peak. In fact, for more than a fifth of days, the 5 pm temperature is higher than for the next 24 hours. This is significant because the NWS recommendation had been to reset at 5 pm.

A more sensitive histogram is of the durations between maxima, shown below for the same reset times. "Normal" is about 24 hours. A short interval, or a long one of near 48 hours, indicates that the same peak is effectively being counted twice. This is very marked at 2pm, but also strong at 5 pm (and significant at 9 am).


I'll show this side-lobe effect as a daily cycle by plotting the variance of the histogram. This increases with the size of the side lobes:



Minima behave somewhat similarly, though with the afternoon effects replaced by morning. Here is the variance plot for the difference between minima. In fact doubling of minima is rarer, perhaps because the minima themselves vary less, or because the daily minimum is less peaky.



Temperature bias


So here is a plot, as a function of reset time, of
  • the three year average max, Tmax
  • the three year average min, Tmin
  • (Tmax+Tmin)/2, the min/max used in indices (in black)
Each is plotted relative to its mean


Each temperature should be adjusted to restore it to a standard reset time. Vose et al quoted above say this should be midnight. This adjustment really only is important if the time of observation changes, introducing an apparent trend. Again Vose et al explain how changes did occur in USHCN. I've also plotted each individual year, just for the min/max, to show that the pattern is fairly reproducible from year to year:
So there is every reason to expect that adjustments calculated from the present hourly obs can be applied to past readings where we only know the min/max and time of obs.

Adjustment

The range shown here is quite large (for this 1 station), about 1.5°F, while the TOBS adjustment in practice was only about 0.3°F over a century. Not all stations did change their time of obs, and those that did typically changed from about 5pm to 9am, which only has a fraction of the full effect.

Thursday, September 20, 2012

Arctic Sea Ice Extent minimum reached



After a couple of days of substantial ice increase, and given the time of the month, I think it is safe to say that the ice extent minimum has been reached. Both IJIS JAXA and NSIDC agree on the date - 16 September. The numbers were:

DateExtent million sq km
IJIS JAXA16 Sept3.489063
NSIDC16 Sept3.36855


IJIS multi-year graph here.

Sunday, September 16, 2012

Javascript rendering of spaghetti plots



My first Javascript exercise used Javascript to enhance visibility of plots with many curves (spaghetti). You could roll the mouse over names in a legend, and in turn the named curves would stand out in black.

I had intended to make more use of that, but somehow that hasn't happened. A minor reason is that the JS code was unnecessarily complicated (being my first).

More spaghetti plots have been appearing at Lucia's, this time re GCM model results, and there was discussion of how to make them more readable. So I thought it was time to scrub up my old code, and this time make the generation of JS an automatic process that others could use.

So I'll give below first the GCM results plots, and then instructions on how to generate it from your own data file.



GCM results

Firstly I should say that I haven't exactly reproduced Lucia's data. I took the files that I used and explained in this post. The various temperature indices are up to 2010 only.

I've followed her lead in doing 13-month filtering. I normalised baselines by setting the mean of each curve over the plot period to zero. This is not part of the program, but was done in creating the csv file.

So here's the plot. Move the mouse over the legend and the named curves will show in black, with the regression line also showed.




How to

The steps are:
  • Download the R code spag.r. Among the first commands are the definition of a name and a url picUrl (end with "/"). Choose your own name and location where the pictures will be held.
  • Create a data CSV file (all in the same directory) called name.csv. You can do this with Excel or other apps, or just by editing. The first column is the x axis values, subsequent cols are the curves. Each col must have a heading (it's name). In CSV, everything is comma-separated, but spacing doesn't matter. Headings (but not numbers) should be in quotes. Use NA for missing.
  • Run spag.r in R. It will produce name.htm and a set of .png files, one for each curve. They are called name1.png etc ("name" of course is the name ypu edited into the file).
  • Move the .png files to the location specified by picUrl.
  • Paste name.htm into your own html file where you want the plot to appear.
For this example, I've posted the CSV file zX.csv.

Saturday, September 15, 2012

August GISS Temp up 0.09°C - ice news


The GISS land/sea monthly anomaly rose from 0.47°C July to 0.56°C in August. GISS has recently been more volatile than other land indices (though similar to satellite), and again this month TempLS was steady. Time series and graphs are shown here

Meanwhile, Arctic ice has been slowly melting. I've added a line on the monitoring page where you can see what minimum will be reached if melting follows the pattern of previous years. The current IJIS JAXA extent is 3,542,000 sq km, which is 749,000 sq km below the previous record minimum in 2007.  The greatest further melting in recent years (2006-) was in 2010; if we follow that pattern it will reach 3,357,000 sq km. IJIS is still melting - NSIDC has shown some refreezing, but that may change again.



As usual, I compare the previously posted TempLS distribution to the GISS plot.
Here is GISS:



And here is the previous TempLS spherical harmonics plot:

.

Previous Months

July
June
May
April
March
February
January
December 2011
November
October
September
August 2011

More data and plots

Monday, September 10, 2012

August TempLS Global Temp up slightly (0.01°C) from July



The TempLS analysis, based on GHCNV3 land temperatures and the ERSST sea temps, showed a monthly average of 0.48°C for August, up from 0.47 °C in July. After a rapid rise to April this year, TempLS has been steady, with a small downward drift. There are more details at the latest temperature data page.

Below is the graph (lat/lon) of temperature distribution for August 2012. I've also included a count and map of the stations that have reported to this date, and a survey of the degree of error caused by my customary early reporting.




This spherical harmonics plot is done with the GISS colors and temperature intervals, and as usual I'll post a comparison when GISS comes out.

And here, from the data page, is the plot of the major indices for the last four months:


Here is the map of "stations" which contributed to this report. Over half are SST readings from the ERSST grid, and generally all of these are available (subject to modification) from the start of the month. But numerous land stations haven't reported. There are a total of 4195; past months with more complete reporting have about 4500.



I was curious to test the effect of incomplete data. I report earlier, and with less complete info, than the majors. This is partly to be first, but also so the calculation is a real test of TempLS's ability to independently assess the average. But I was curious to see what the penalty was. It isn't much. Here is a table of the difference between the originally posted monthly averages for 2012 and the current values, including the extra data that has come in since.

JanFebMarAprMayJunJul
0.0180.0090.0280.000-0.0050.005-0.003


Sunday, September 9, 2012

Cold Confusion


Eli has noted a rather remarkable scientific riff from Mitt Romney:
I do believe in basic science. I believe in participating in space. I believe in analysis of new sources of energy. I believe in laboratories, looking at ways to conduct electricity with -- with cold fusion, if we can come up with it. It was the University of Utah that solved that. We somehow can’t figure out how to duplicate it.

Well, of course Romney doesn't claim to be a scientist. It sounds like he's mixing up cold fusion with high temperature superconductivity. And I guess the last sentence is true.

Anyway, I was curious to track down the source and context. It's from an interview Dec 7 2011 with the Washington Examiner. The transcript and video are here. It's on video part 2 (actually audio only) at about 5 min.

Tuesday, September 4, 2012

Empirical Model - WUWT Style


At WUWT, Girma Orssengo, PhD, has a guest post showing some of the empirical models that he often posts in comments at Climate etc and elsewhere. It shows a parabolic+sinusoid fit to HADCRUT3:



The key plot is the one on the right. It's often been observed that a 60-year (or thereabouts) cycle plus a linear or quadratic does fit in this way. The question is, what does it mean? And in particular, can it be used for prediction?

Girma says that it isn't meant as a basis for physics interpretation. But when challenged about prediction, he said:

" Girma says: September 4, 2012 at 12:53 am

Nylo
A model is only useful if it allows predicting future behaviour. I see no predictions by the author that could be later falsified by the real outcome.

The model established a pattern as shown in Graph “f”. From this graph, It is easy to predict the climate if the pattern continues => Little warming in the next 15 years."


So prediction does seem to be an aim.

The problem with curve fitting is that behaviour outside the range is determined by the rather arbitrary selection of basis functions. I tried a regression fit in R using the same basis functions, with similar results:




Then I tried the same sinusoid, a 60-year cos function with min at 1910, but instead of t and t2 I used exp(t/20) and exp(t/100):



The fit is very similar, and it's not easy to declare that one is better than the other. Here they are on one plot:



But they give very different predictions. The divergence looking forward to 2050 is not so huge:



But looking forward to 2100, the exponential model (green) runs right away, warming by 8°C. Grim.



The bind with fancy curve fitting is that it really can't be used much outside the range. And within the range, it isn't telling you anything you didn't already know, unless you are confident that you can make a useful deduction about the physics.

So what about linear regression - very commonly used? Well, it isn't "fancy" curve fitting. But the underlying principle is there. If it is a time series, you need to be open to the possibility that there is a real underlying rate of change. Then regression (curve fitting) can estimate the rate for you. But it doesn't prove that the linear model is right.