It's unfortunate that Roy doesn't seem to give any trends from ISH that aren't subject to his population density adjustment, because then we could compare like with like. However, I will look at the GHCN unadjusted data for ConUS. This is a real test of whether the WUWT (and RS) claim that the trend is due entirely to adjustment is justified. It isn't much of a test if you strike their adjustments and then introduce a big new one.
I actually looked at ConUS with an early version of TempLS here. I looked at the period 1978-2009, and got a trend of 0.255°C/decade. This was using GHCN v2 min/max mean monthly data, with weighted regression using 5x5° cells. I thought I would update this calc, and for the period used by Roy. With TempLS V2.2, GHCN
Along the way, I discovered a trick question. What is the easternmost US GHCN Met station?
The answer is Shemya AFB in the Aleutians, Longitude 174° East. That did trick me - in TempLS you provide logical statements to specify subsets, and I thought for ConUS requiring country code 425 (US) and east of Long 125W would do it. So Shemya, and another station at Attu, got included. There weren't many readings, and I don't think they changed the trend by much.I've fixed it now.
Station distribution bias
Anyway at WUWT Willis Eschenbach asked what precautions I had taken to avoid E coast bias. Probably the first thing to say is that GHCN has taken precautions. They have been criticised for pruning their list of stations but one thing that has been achieved is a reasonable uniformity. Here's a map of the stations reporting in 2011, and also all stations reporting at some point since 1972: |  |
The distribution isn't that uneven. However, Ver 2.2 has some capabilities in mesh-based weighting. I was experimenting with those in Antarctica, and so I saw a chance to try them out here.
Weighting
The first thing I should say is that I haven't generally found that messing with weighting schemes changes average trends a lot. I should emphasise that the weighting scheme doesn't change the data, only the way it is included. Reweighting one station relative to another only has a big effect if they are very different. And of course, the changes tend to balance out in the average.The purpose of weighting is essentially to form a sum which is a good approximation to a space integral. This has the appropriate physical significance, and also gives a rational basis for saying that bias due to clustering of stations has been overcome.
The best way I have found to use meshes is to surround each node of the mesh (which is done for every month, so the nodes are the stations reporting in the month) by the area formed from the adjacent portions of the triangles marked out by the medians and the edge bisectors. The preferred method is Voronoi, which forms the in-centres and the perpendicular bisectors. But here we're only trying to associate with each node an area for weighting purposes, so it isn't critical to have the areas exactly demarcated by closest neighborhood, as long as they are close and unique. And this scheme is a lot faster, which is important when you need a new tesselation every month over 40 years.
Convexity
I generate the meshes using R's convhull function. I use the 3D version, which means that I find the hull of the curved area (US). There is a back mesh, which is fairly easy to get rid of because the normals face the other way. But for a country shape, it incorporates bays etc with the land area.For Antarctica I had trouble with the Weddell Sea and used a scheme for carving part of it out. This time I tried a new scheme which gets rid of the back mesh as well. I introduce a ring of new points (before meshing) which form a ring outside the stations. The idea is that if you ran a string around the surrounding nodes, it would be clear of any real stations. Then after having meshed the larger area, I eliminate the extra nodes and any triangles entailed. That tends to strip down to the real country outline. It's worth reminding that TempLS doesn't actually know about borders - it only has the location of the selected stations.
Meshes and weights
So here is the result. I'm showing the mesh in the background and weights proportional to the area of the circles around each station reporting in a particular month.
You can see that stations in sparser areas are upweighted according to the area they are taken to represent.
Trends
With the most recent mesh as shown in the diagram, I found that the trend using unadjusted GHCN monthly data for ConUS stations was 0.167°C/decade from 1973 to 2011. This is very similar to the cell-weighted figure (0.161), somewhat less than CRUTEM 3 (0.198) and less again than USHCN (0.245). Notwithstanding, I believe that the USHCN adjustments are justified, and theirs is the preferred figure. But is certainly isn't true that the adjustments are responsible for the uptrend.More on adjustments
At WUWT Geoff Sherrington made the predictable objection that GHCN unadjusted data has been tampered with by the met stations. I've seen this often, but never with evidence.In fact the original GHCN (mid 90's) was compiled direct from the records, to the extent that they preserved the various fragments from different sources. They then distributed that on CD, to many recipients.
In more recent times, I have never seen any indication that GHCN unadjusted records have been adjusted. There are occasional records removed when errors are found. Others come in late and are included. But they say that they don't modify records except on receipt of an amended CLIMat form from the Met supplier, and that seems to be true. Since the mid 90's these forms have been used exclusively, and you can inspect them here.
Posts
We've seen all this before. Don't you ever get bored of reading and interacting with those crowds?
ReplyDeleteI guess we're back to the phase of the moon where they're all saying there hasn't actually been any warming. Wasn't all that long ago that they were protesting that they never said that. Somebody should figure out the periodicity of the varying arguments.
Well, there are lots of people reading, so re-enacting the same old arguments isn't entirely worthless. But I agree about the lack of progress. As Phil Clarke pointed out, 15 hours ago, the graphic which cites the adjustment slopes as if they were Celsius (though they were Fahrenheit) is from a 2007 post. The false claim was criticised then, but here it is, reappearing as if nothing had happened. And still uncorrected.
DeleteUsing the fact that the explanation for the NOAA graph incorrectly puts the values in Celsius instead of Fahrenheit to infer that climate change crisis skeptics are being disingenuous and not making any progress, is really very lame. It would be akin to me proclaiming that the misspelling of 'criticized' in your comment above invalidates your arguments.
DeleteThe truth is that both errors have nothing to do with the arguments being presented.
misspelling of 'criticized'
DeleteWell, in my part of the world, criticised is correct. But what's the argument here? That getting the units wrong doesn't matter? The numbers don't matter? Then why cite them?
The WUWT headline was "Warming in the USHCN is mainly an artifact of adjustments". When you're making such claims, a factor of 1.8 matters. But it's the repetition and recycling of clearly wrong facts that grates. It's now over 30 hours since Phil Clarke pointed out the error, with no acknowledgement. I also haven't seen any response from Roy to the observation that his own UAH USA48 has a trend very similar to USHCN.
I saw the C/F confusion there a couple years back, too - it became quite predictable that you'd see that confusion when the topic of US adjustments came in.
ReplyDeleteOne thing that bothers me a little is whether we know how much TOB might be in the non-US stations.
I think the unique thing about the US was the volunteer COOP observers. Mostly in ROW observers were employed and read (or at least logged the time) when they were told, which was at least consistent. In the US the NWS had advised reading rain in the morning and temp in the evening. But observers increasingly found one reading a day easier, and settled for morning.
DeleteYou know, it's amazing to read the Orwellian doublethink over there. As pointed out, the UAH satellite analysis finds substantial warming over the US. OK, that's the troposphere, not surface air temperature. Except Watts OWN study (Fall et al. 2011) finds substantial warming in the USHCN, but according to Watts, Spencer's blog post "proves what we have been saying for years." And how do we know that "warming in the USHCN is mainly an artifact of adjustments?" By adjusting the data of course!
ReplyDeleteSo, to review. Patriots adjust for UHI and communists adjust for station moves, instrument changes and TOBS. And if anyone points out that unadjusted data also shows strong warming, then we must question whether the unadjusted data is actually unadjusted, because, you know, communists might have infiltrated those COOPs.
So, who is actually going to bother read Woy's stuff carefully enough to actually work out where the error in the population-causes-temperature stuff is?
ReplyDeleteThat's Dr. Woy PhD, to you.
DeleteBelette "So, who is actually going to bother read Woy's stuff carefully enough to actually work out where the error in the population-causes-temperature stuff is?"
DeleteThat's so adult. .
I did just read his Apr 5 post (see below). It seems to set out the method most explicitly. It's just a regression of trend against PD^0,2. It's a poor-looking fit to me, but he claims the slope has se of 20%. Then he just subtracted the amount of trend predicted for each population density by the regression.
DeleteApart from the dodginess of the curve fitting, there is a clear error there. He's correcting to zero PD. But the PD of ConUS isn't zero. It's 40/sq km. And reading from his graph, the predicted trend for that PD is 0.12 C/decade. IOW, the right PDA value would be 0.12+0.013=0.133 C/year. Not so different from his unadjusted value.
Hopefully you know I was commenting on Belitte's use of name calling.
DeleteI don't take Roy very seriously in this stuff. I'm glad he's not the one doing the coding for UAH ( Even then, sub-annual variation in that series is easily shown to be pure garbage (it's results from the problem of subtracting poorly modeled large systematics IMO).
I do admire your willingness to spend as much time on WUWT and in dealing with Roy's meandering arguments as you do. (I don't have any problems with the way you critique him, you manage to stay a lot more impeccably polite than my irascible self manages when dealing with idiocy.)
I think the post should have been withdrawn after so many people pointed out problems on it on WUWT. Oh well, at least there's no dragon slayer haven there....
Carrick - yes, I was responding to the substance of Belette's comment rather than to you - that's the constraint of the threading. And to be fair to Roy, he wasn't responsible for the C/F stuff. It's not the first time a WUWT post has been undermined by bungles in AW's intro. Still, the post had plenty of problems in its own right.
DeleteOK, good. I never noticed any substance in Belette's comment, but maybe that's just me reading less into what he said than what he meant. >.>
DeleteA fun statistic there. As I said, Roy corrects to a US with no inhabitants. If it ever did reach that situation, then on Roy's 1/5 power model, the entry of just one inhabitant would raise ConUS temperature trend by 0.0023C/decade.
DeleteThe major problem with the entire series of posts is that local (county level and lower) PD in the US has changed radically in the past 40 years, with major sections of the midwest west of the Mississippi decreasing by 50% and CA and other places growing by huge amounts. That means any correlation against anomalies in one year are useless.
DeleteUnless it's accompanied by reverting urban areas to forest, you could argue that decreasing PD is the same as constant PD. But regardless of that, Dr Roy PhD has to put some more thought into the physical meaning of the arithmetic games he is playing.
DeleteThe Fell, et al, paper (Pielke Sr.) has an interesting comparison of land use (changes) and the GISSTemp urban/rural categorization which shows that at least for the US these match up pretty well. As far as major land use changes, the reforestation of the US East after 1900, if Spencer is right, should have completely driven the temperature anomaly changes down into the earth.
DeleteStill, if you are arguing about temperature anomalies in terms of population density, then you have to look at the population density anomalies. These were MAJOR changes even over 30-40 years (see RR for the maps). Even Eli was surprised.
just wondering that some of the circles are very much larger than others...is there a check on the homogeneity of the readings within each circle? Apologies if you covered this already in 1974
ReplyDeleteGraeme,
DeleteIt reflects the irregular distribution of stations. The intention is that stations in sparse areas should get big weights. But there is also variation due to the way triangular meshing works out. The weight is proportional to (1/3 of) the total area of the triangles of which the station is part. A station can have mostly small triangles around, but also be attached to a large one extending into a sparsely covered area, and will thus get a bigger weight than its neighbors.
The local uniformity of irregular meshes has been much studied in finite element analysis. I've taken a mesh from R that could probably have been improved. But the penalty here for lower quality is much less than in FEA - a small increase in the variance of the estimate.
I said above that Roy should have given ISH results without his PDA, so that like could be compared with like. I see now that he has done this, somewhat indirectly, in his April 5 post. He doesn't give the actual number, but says it is 20% less than CRUtem 3, which I guess makes it about 0.16C/decade - almost exactly what I got. I think that if he's claiming that USHCN adjustments make up all the trend, then this fact should be stated more directly. It means that rather, the PD adjustment takes away most of the trend. A rather different proposition.
ReplyDeleteHe also deals there, in a way, with the contradiction between his claim of no trend and the 0.22 C/decade trend given by his own TLT data. Oddly, his preferred explanation seems to be that the satellite is wrong.
I see that he did state the ISH no-PDA figure at one point; it is 0.157°C/decade. My figure was 0.161.
Delete"Oddly, his preferred explanation seems to be that the satellite is wrong."
DeleteI seem to remember some people being quite happy and pleased with their satellite-based results, before various errors were identified. So I don't know that this is so odd.
Do you have the code posted anywhere for the mesh that you're using in this post. I'm curious to know how practical it is for further use in other study areas. I believe you're aware of mine and Mosher's discussions regarding a study area I'm working on.
ReplyDeleteRobert,
DeleteIt's version 2.2. I haven't released the code yet, and I don't guarantee that the code I used here would work for everything. But the draft version I used is here.
Along the way, I discovered that I had mistakenly used a GHCN V2 file instead of V3. Mostly it doesn't make much difference, but V3 includes a lot of USHCN stations that had been removed from V2 in recent years. This makes a minor difference to the trend (0.168 instead of 0.161) but a major difference to the appearance of the plot of mesh for June 2009, which is much more crowded in V3 (and quite hard to follow).
To run the code, just unzip in one directory and type in R
jobname="US"
source("TempLSv2.2.r")
Results will appear in a file called "US".
Nick's mesh stuff looks cool, Robert. Once I make some more progress on this pesky 'merge" process, I'll have a look at it.
DeleteThis comment has been removed by the author.
ReplyDeleteHullo, Nick, we met in Lisbon a little over a year ago, and since then I have been lurking regularly on your site and also following your tireless struggles on WUWT and other AGW-hostile blogs. Have you ever had an experience when one of the denizens of such sites responds to one of your postings with a reply along the lines of: "thanks for that, Nick, I realise that I was wrong", as opposed to the usual obfuscations along the lines of "yes, but you haven't answered my point on humidity".
ReplyDeleteBut to return to the point at issue, since you have gone to the trouble of reading Dr Roy PhD's postings on the subject have you worked out the justification for taking the fifth root of population density, or is this just another of Dr Roy PhD's arbitrary peregrinations through through parameter space. I'm afraid I'm too lazy to look for the answer myself.
Bill,
DeleteGood to hear from you again! And that is a good and important question. Looking at his plot, it seems to me that you would be hard put to discern a trend, let along say what the curvature is, which is what is needed to determine the power.
But it's very important to his estimate. If you look at his line fit, it drops about 0.059°C/decade between PD 1 and zero. He corrects to zero, so that is another 0.059 that he deducts from all trends. But he has no data at all below 1. That extra 0.059 is based purely on the form of his postulated curve. If he'd chosen power 0.1, he'd have a fitting curve that would be indistinguishable in fit quality, but would double this interval.
As I mentioned above, the fact that he has zero stations with PD below about 0.9 simply reflects the fact that that is a very low pop density for the US. The ConUS average is 40, and on that basis his average "corrected" trend would be about 0.13 C/decade. RS might argue that the mean is overweighted by big cities and a median might be better. But even halving the PD would still give a trend of about 0.11.