Its worth noting the the breakpoints are detected by a comparison to nearby stations. Those stations that show irregularities post-homogenization are likely cases where the irregularities show up in nearby stations as well (and may actually be a signal). Similarly, inhomogenities in stations with no nearby counterparts will generally not be detected.
Nick, good to hear you have done this. I started writing a code to look through all the stations systematically but it's quite fiddly and I haven't had time. I disagree with your final point. No reading is required to see that the Iceland adjustments are wrong.
I think Zeke needs to read the previous comments about the Iceland adjustments. There was a sharp cooling in 1965 in all the Iceland raw data sets. This was deleted in almost all cases by the erroneous GHCN adjustment algorithm.
Zeke, Yes, and that would affect the interpretations here. But I think you do have to have a station irregularity first - then it is checked with neighbors.
One of the objections raised about Iceland was that in about 1965 several of them were corrected in a similar way. It wasn't easy to see that as a neighbor-forced thing. I think that is what Paul is referring to.
Paul, I don't know if R code helps but I'll try to scrub mine up and get it online.
Paul, My point is, though, that a statistical evaluation is what is needed. There is a cost in noise to correcting apparent bias, as here. This may well be a false positive. The objective is to ensure that the noise introduced is less harmful (biased) than the noise removed.
I meant that neighbors are used to detect irregularities in the first place (and later correct for them). Metadata is also used if available, though it simply reduces the threshold for breakpoint detection rather than forces a breakpoint, given that metadata can be wrong.
PaulM,
I was simply pointing out how the Menne and Williams algorithm detects breakpoints. I'll read over your prior Iceland comments, but if multiple stations in the region show the same behavior at the same time it is unlikely that they will be corrected.
Zeke, this is exactly what the fuss is about! If multiple sites in the same area show the same behaviour, then clearly they shouldn't be "corrected". But this is exactly what happens in the GHCN adjustments for Iceland. As well as the 1965 issue, there is the consistent warm period around 1940 where temperature was similar today, apparent in all the raw data and in the literature. But the GHCN adjustments put in a cooling here. See the sequence of posts at Paul Homewood's blog. There is something wrong - either the adjustment algorithm is 'over-zealous' or there is a coding error.
I have the Lawrimore et al paper if anyone wants a copy.
GHCNM v3.1.1 is released with changes in scripts due to asynchronous execution problems. Also added ability to add new stations automatically when their period of record increased enough to process and improved testing when not enough neighbors to estimate adjustments.
I'm not sure what 'asynchronous execution problems' means. Anyway the adjustments are now significantly different, but no better. The erroneous 1965 adjustment now appears in all 8 iceland stations (before it was only 7/8) and the fabricated warming in Reykjavik is worse. But some stations eg Stykkisholmur are better.
The warming adjustment at Corona NM is now "only" about 4 degrees rather than 5. Can you download the new dataset and run your code again to see where the biggest adjustments are now?
Anon, Yes, the adjustment does have some upward effect, and the updated algorithm (next post) raises it a bit more. But it's nothing like the impression you get from these extreme examples.
Its worth noting the the breakpoints are detected by a comparison to nearby stations. Those stations that show irregularities post-homogenization are likely cases where the irregularities show up in nearby stations as well (and may actually be a signal). Similarly, inhomogenities in stations with no nearby counterparts will generally not be detected.
ReplyDeleteNick, good to hear you have done this.
ReplyDeleteI started writing a code to look through all the stations systematically but it's quite fiddly and I haven't had time.
I disagree with your final point. No reading is required to see that the Iceland adjustments are wrong.
I think Zeke needs to read the previous comments about the Iceland adjustments.
There was a sharp cooling in 1965 in all the Iceland raw data sets.
This was deleted in almost all cases by the erroneous GHCN adjustment algorithm.
PaulM
Zeke,
ReplyDeleteYes, and that would affect the interpretations here. But I think you do have to have a station irregularity first - then it is checked with neighbors.
One of the objections raised about Iceland was that in about 1965 several of them were corrected in a similar way. It wasn't easy to see that as a neighbor-forced thing. I think that is what Paul is referring to.
Paul, I don't know if R code helps but I'll try to scrub mine up and get it online.
Paul,
ReplyDeleteMy point is, though, that a statistical evaluation is what is needed. There is a cost in noise to correcting apparent bias, as here. This may well be a false positive. The objective is to ensure that the noise introduced is less harmful (biased) than the noise removed.
Nick,
ReplyDeleteI meant that neighbors are used to detect irregularities in the first place (and later correct for them). Metadata is also used if available, though it simply reduces the threshold for breakpoint detection rather than forces a breakpoint, given that metadata can be wrong.
PaulM,
I was simply pointing out how the Menne and Williams algorithm detects breakpoints. I'll read over your prior Iceland comments, but if multiple stations in the region show the same behavior at the same time it is unlikely that they will be corrected.
Zeke, this is exactly what the fuss is about! If multiple sites in the same area show the same behaviour, then clearly they shouldn't be "corrected". But this is exactly what happens in the GHCN adjustments for Iceland. As well as the 1965 issue, there is the consistent warm period around 1940 where temperature was similar today, apparent in all the raw data and in the literature. But the GHCN adjustments put in a cooling here. See the sequence of posts at Paul Homewood's blog.
ReplyDeleteThere is something wrong - either the adjustment algorithm is 'over-zealous' or there is a coding error.
I have the Lawrimore et al paper if anyone wants a copy.
Paul
A noted example occurred in late 2009, when Willis Eschenbach became excited about GHCN V2 adjustments to the temperature at Darwin.
ReplyDeleteJust wait until he discovers the upward trend adjustment at WILLIS ISLAND (off the northeast coast of Australia).
Ned
Nick,
ReplyDeleteGHCN have changed their version!
A comment in the Changelog file says
********************************************************************************
02/07/2012
GHCNM v3.1.1 is released with changes in scripts due to asynchronous execution
problems. Also added ability to add new stations automatically when their
period of record increased enough to process and improved testing when not
enough neighbors to estimate adjustments.
********************************************************************************
I'm not sure what 'asynchronous execution problems' means.
Anyway the adjustments are now significantly different, but no better.
The erroneous 1965 adjustment now appears in all 8 iceland stations (before it was only 7/8)
and the fabricated warming in Reykjavik is worse.
But some stations eg Stykkisholmur are better.
The warming adjustment at Corona NM is now "only" about 4 degrees rather than 5.
Can you download the new dataset and run your code again to see where the biggest adjustments are now?
Paul
Thanks for the note, Paul. Yes, I'll re-run.
ReplyDelete. . . Mean 0.0196 °C/Decade . . .
ReplyDeleteThat's a tad high, if you've only got +0.7°C over the last century or so. I think they're over-adjusting there.
Anon,
ReplyDeleteYes, the adjustment does have some upward effect, and the updated algorithm (next post) raises it a bit more. But it's nothing like the impression you get from these extreme examples.
The thing is, it may be right.