tag:blogger.com,1999:blog-7729093380675162051.post5870852268458590670..comments2024-03-28T13:56:47.604+11:00Comments on moyhu: Reykjavik and GHCN adjustments.Nick Stokeshttp://www.blogger.com/profile/06377413236983002873noreply@blogger.comBlogger17125tag:blogger.com,1999:blog-7729093380675162051.post-60316600394228898672012-02-06T05:59:51.695+11:002012-02-06T05:59:51.695+11:00Mailman,
"when ever adjustments are made the...Mailman,<br /> <i>"when ever adjustments are made the effect is to always cool the past "</i><br />That's just not true. In the <a href="http://www.moyhu.blogspot.com.au/2009/12/darwin-and-ghcn-adjustments-willis.html" rel="nofollow">Darwin post</a> that I linked, I showed the <a href="http://www.gilestro.tk/wp-content/uploads/2009/12/dist1.png" rel="nofollow">histogram</a> of effects of V2 adjustments on trend. It's balanced; the trend is almost as likely to be lowered as raised.<br /><br />That post was a response to a WUWT post on adjustments to Darwin, where adjustments raised the trend. I showed that <a href="http://1.bp.blogspot.com/_PkG6PPXjYgI/SzCkGKum6eI/AAAAAAAAAAM/Hs8pCsnD8bs/s640/GHCNCoona.jpg" rel="nofollow">adjustments to Coonabarabran</a>, also in Australia, lowered the trend there equally.<br /><br />It's true that adjustments don't change the present. That's the reference point.Nick Stokeshttps://www.blogger.com/profile/06377413236983002873noreply@blogger.comtag:blogger.com,1999:blog-7729093380675162051.post-55873375586203527012012-02-06T02:56:51.120+11:002012-02-06T02:56:51.120+11:00Could someone explain why it is that when ever adj...Could someone explain why it is that when ever adjustments are made the effect is to always cool the past while not touching the present?<br /><br />Regards<br /><br />MailmanAnonymousnoreply@blogger.comtag:blogger.com,1999:blog-7729093380675162051.post-42256891478783262862012-02-03T05:51:13.604+11:002012-02-03T05:51:13.604+11:00NickB, look at the dropdown options and download t...NickB, look at the dropdown options and download the data for Reykjavik for option 1 and option 3. You can see that GISS is still dong its own homogeneity adjustment after the GHCN one (in fact somewhat correcting the GHCN, as I said above). <br /><br />PaulMAnonymousnoreply@blogger.comtag:blogger.com,1999:blog-7729093380675162051.post-44124026927739618412012-02-03T05:45:51.714+11:002012-02-03T05:45:51.714+11:00Nick B, perhaps you haven't been following thi...Nick B, perhaps you haven't been following this story. <br />I recommend you go and look through the posts at Paul Homewood's blog. <br />In all 8 of the iceland stations unadjusted data, there is a sharp drop around 1965, and this is well established from a number of sources. <br />In 7 of the 8 cases the GHCN adjustment algorithm incorrectly puts in a break here and gets rid of this sharp drop. <br />Similarly, the raw data consistently shows a warm period around 1930-1940 which is also established in the literature on air temps and SST temps (papers by Hanna et al). In most cases the GHCN algorithm adjusts these downwards. <br />So whatever is flagging the GHCN adjustments, it's not nearby neighbours. If the algorithm looked at neighbours in any sensible way it would know that the raw data was valid.<br /><br />Here's what the Iceland Met Office says:<br /><br />' The GHCN "corrections" are grossly in error in the case of Reykjavik but not quite as bad for the other stations. But we will have a better look. We do not accept these "corrections". '<br /><br /><br />Of course if they would publish the code, the error could be found fairly easily. <br /><br /><br />PaulMAnonymousnoreply@blogger.comtag:blogger.com,1999:blog-7729093380675162051.post-40827813353588577052012-02-02T12:50:25.074+11:002012-02-02T12:50:25.074+11:00Nick,
I'm planning a post which will show the ...Nick,<br />I'm planning a post which will show the V3.1 histogram, coupled with a Google Maps <a href="http://www.moyhu.blogspot.com.au/2011/12/google-maps-display-of-ghcn-stations.html" rel="nofollow">visualization</a> so you can see which stations have large positive or negative adjustments. It should show up things like this Iceland pattern.Nick Stokeshttps://www.blogger.com/profile/06377413236983002873noreply@blogger.comtag:blogger.com,1999:blog-7729093380675162051.post-73088225574199547742012-02-02T07:01:30.582+11:002012-02-02T07:01:30.582+11:00As for GISTEMP, until recently it was using unadju...As for GISTEMP, until recently it was using unadjusted GHCN data. I gather that in December it was changed to use the adjusted GHCN data instead of doing its own homogeneity adjustments. That's what it says <a href="http://data.giss.nasa.gov/gistemp/updates_v3/" rel="nofollow">here</a> anyway. The code is <a href="http://data.giss.nasa.gov/gistemp/sources_v3/" rel="nofollow">here</a> if you want to check.Nick Barneshttps://www.blogger.com/profile/00057838251997644583noreply@blogger.comtag:blogger.com,1999:blog-7729093380675162051.post-16033067565667445952012-02-02T06:57:25.491+11:002012-02-02T06:57:25.491+11:00PaulM: the GHCN algorithm is all about neighbourin...PaulM: the GHCN algorithm is all about neighbouring station records. That's how it works. I recommend you read the papers on it.Nick Barneshttps://www.blogger.com/profile/00057838251997644583noreply@blogger.comtag:blogger.com,1999:blog-7729093380675162051.post-54639787636242696322012-02-02T06:43:25.635+11:002012-02-02T06:43:25.635+11:00Nick (S), thank you very much for posting up the v...Nick (S), thank you very much for posting up the v2 file from Dec 2009.<br /><br />Looking at the data for Reykjavik shows that the v2 adjustments cool by about 0.9 in the 60s and 70s, increasing to 1.3 before 1940 and 1.7 before 1920. <br />So the adjustment-fabricated warming in v2 is similar in magnitude to that shown in your graph for v3.1, though more monotonic. It is a similar picture for Stykkisholmur, with downward adjustments of around a degree before 1970.<br /><br />So to recap, we have fairly monotonic past-cooling adjustments in v2, no adjustment in v3.0, and then very erratic adjustments in v3.1.<br /><br />Incidentally I also found some sign errors in the v2 adjustments for Stykkisholmur.<br />The unadjusted file for the first few months of 1973 and 1974 are <br />1973 22 -34 4 21<br />1974 -1 -11 26 54<br />and the adjusted file is<br />1973 -22 -34 -4 21 <br />1974 -1 -11 -26 54 <br /><br /><br />Regarding breakpoints, it would, as you say, be sensible if the algorithm checked nearby sites to determine whether an apparent break was genuine or spurious. It's pretty clear that the GHCN algorithm doesn't do this, since the genuine cooling in 1965 is erroneously adjusted away by the GHCN algorithm in 7 of the 8 icelandic stations. <br /><br /><br />Regarding the GISS adjustments, the current situation is absurd. GISS starts from the GHCN adjusted data, which includes (erroneous) adjustments for breaks and homogeneity adjustments. It then applies its own algorithm to remove 'suspicious records', and then applies its own homogeneity adjustment!<br />It's amusing to see that the GISS homogeneity adjustment puts back some of the 1940s warmth that was deleted by GHCN (0.7 degrees in 1940). <br /><br /><br />PaulMAnonymousnoreply@blogger.comtag:blogger.com,1999:blog-7729093380675162051.post-3376088569718837562012-02-02T05:27:20.445+11:002012-02-02T05:27:20.445+11:00Peter is right, of course, and I totally support h...Peter is right, of course, and I totally support his continuing efforts to develop homogenization testbeds/platforms/systems, which will allow us to improve this whole area of the science. I'm hoping to spend some time thinking about this in February (and revisiting Dan's code, if I have the time).<br /><br />I'm happy to be reminded of <a href="http://moyhu.blogspot.com/2009/12/darwin-and-ghcn-adjustments-willis.html" rel="nofollow">this</a> excellent piece of work, which showed very clearly for GHCN v2 that there is a fairly symmetric (and fairly narrow) spread of trend effects due to homogenization, with very nearly zero net effect on global trends. I'd be glad if you would retread that for GHCN 3.1.<br /><br />Any homogenization process is bound to have trend effect outliers: stations for which the homogenization has a comparatively large effect on the long-term trend. Because the data is so noisy, it is inevitable that some of the adjustments producing those outliers are 'incorrect': that is, the station record at a change point reflected ground truth and not an inhomogeneity. Those are the eggs we need to break to make the climate omelette. Overall studies of homogenization processes, like the excellent recent Williams/Menne/Thorne paper in JGR-A, show very clearly that they are valuable in recovering climate signal from inhomogeneity noise.Nick Barneshttps://www.blogger.com/profile/00057838251997644583noreply@blogger.comtag:blogger.com,1999:blog-7729093380675162051.post-83167195034936036632012-02-02T02:32:37.407+11:002012-02-02T02:32:37.407+11:00I left a trailing html tag somewhere in the above....I left a trailing html tag somewhere in the above. Sorry. My bad for writing quickly and not checking.PeterThornehttps://www.blogger.com/profile/02438826461353615177noreply@blogger.comtag:blogger.com,1999:blog-7729093380675162051.post-73259260272893769432012-02-02T01:48:58.386+11:002012-02-02T01:48:58.386+11:00I find this wailing and gnashing of teeth somwewha...I find this wailing and gnashing of teeth somwewhat counter-productive. We need to start from a realization that none of these measures were made in a metrologically traceable manner to SI standards, that change has been ubiquitous even at the best managed sites, and so adjustments are required. In the absence of complete metadata as to changes at each station there will be inevitable uncertainty in how to do the adjustments. For climate trends this is the elephant in the room - because (lack of) adjustments yield red-noise changes which have increasing influence with trend length. Key is to get the adjustments right. There are ways that we can do this better than we have in the past see e.g. <a href="http://surfacetemperatures.blogspot.com/2012/01/benchmarking-and-assessment-applied-to.html" rel="nofollow">this</a> recent effort to benchmark the performance of the algorithm in question over the CONUS sub-domain. But what we really need is more efforts to look at the adjustments problem which is <b>the</b> problem that matters from a long-term change perspective. Then we need a consistent set of benchmarks. More on this at <a href="javascript:void(0);" rel="nofollow">BAMS</a> article.PeterThornehttps://www.blogger.com/profile/02438826461353615177noreply@blogger.comtag:blogger.com,1999:blog-7729093380675162051.post-4001759857559652122012-02-01T22:14:56.337+11:002012-02-01T22:14:56.337+11:00Indeed, I put that blog post up myself. Note that...Indeed, I put that blog post up myself. Note that the code which Dan started with was apparently not exactly the same as the code which was being used at the time (last summer) for GHCN 3.0. Matt & Claude had already made some improvements, which then combined with fixes for bugs identified by Dan, for the 3.1.0 code (the process is described in their tech note). See <a href="http://surfacetemperatures.blogspot.com/2011/11/ghcn-m-v310-showing-value-of-engaging.html" rel="nofollow">here</a>.Nick Barneshttps://www.blogger.com/profile/00057838251997644583noreply@blogger.comtag:blogger.com,1999:blog-7729093380675162051.post-9691616529680104042012-02-01T21:42:33.900+11:002012-02-01T21:42:33.900+11:00Thanks, Nick, I didn't know that. It's in ...Thanks, Nick, I didn't know that. It's in the <a href="ftp://ftp.ncdc.noaa.gov/pub/data/ushcn/v2/monthly/software/" rel="nofollow">USHCN directory.</a> I've downloaded it; there's a lot of documentation.<br /><br />Daniel Rothenberg's project is described <a href="http://climatecode.org/activities/gsoc2011/rothenberg/" rel="nofollow">here</a>.Nick Stokeshttps://www.blogger.com/profile/06377413236983002873noreply@blogger.comtag:blogger.com,1999:blog-7729093380675162051.post-79324255687769986132012-02-01T10:39:58.474+11:002012-02-01T10:39:58.474+11:00The GHCN 3.0.0 code (Menne & Williams 2009) wa...The GHCN 3.0.0 code (Menne & Williams 2009) was available from the NCDC FTP site; that's where Dan Rothenberg got it to do the ccf-homogenization work last summer. The 3.1.0 code is not available yet. My understanding is that researchers need to get some sort of permission to post code on the internet. Bureaucracies can be like that - possibly a hangover from NCDC's military past? Who knows. Anyway, I believe that the code will soon appear.Nick Barneshttps://www.blogger.com/profile/00057838251997644583noreply@blogger.comtag:blogger.com,1999:blog-7729093380675162051.post-54436592404714407562012-01-31T19:28:52.396+11:002012-01-31T19:28:52.396+11:00Looks like the unadjusted versions are still avail...Looks like the unadjusted versions are still available via gistemp's station selection page. That is: change the url to the old format for the "data_set"-parameter; 1 or 2. 1 is unadjusted I believe.<br /><br />http://data.giss.nasa.gov/cgi-bin/gistemp/gistemp_station.py?id=620040300000&data_set=1&num_neighbors=1<br /><br />When choosing gistemp's homogeneity adjustement this is the result:<br /><br />http://data.giss.nasa.gov/cgi-bin/gistemp/gistemp_station.py?id=620040300000&data_set=14&num_neighbors=1<br /><br />More like the unadjusted...Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-7729093380675162051.post-76453161046279747422012-01-31T06:40:27.522+11:002012-01-31T06:40:27.522+11:00Paul,
I don't think Hadcrut has used the GHCN ...Paul,<br />I don't think Hadcrut has used the GHCN adjusted data in the past. Maybe they will.<br /><br />I have uploaded the zipped v2 Dec 2009 adjusted file <a href="https://s3-us-west-1.amazonaws.com/www.moyhu.org/GHCN/v2Dec2009.zip" rel="nofollow">here</a>. I didn't include the unadjusted file, or v2 inventory, because they don't change. But I can if you need them.<br /><br />Homogenization doesn't interact with quality control, which is mainly about finding clerical errors. As you say, the importance of the file is that it is used by GISS. But the test to apply for that is statistical - do the adjustments create a bias? And if so, is that justified?<br /><br />But I agree that here it is probably interpreting the sudden changes in 1940 and 1965 as extraneous. As I said at WUWT, it would be a big mistake to step in and overrule the algorithm piecemeal. The only option is to make a general rule modification and apply it everywhere. For example, make it reluctant to spot a break if it is shared by neighbors. I suspect they do that already.<br /><br />I don't think the GHCN adjustment code is available for download. The GISS code is, and has been updated recently; I suspect it will show whether GISS has scrapped its own process in favor of GHCN. I'm checking.<br /><br />I think an aspect that has bothered Ole Humlun, and maybe others, is the GISS interface for station data. It used to have unadjusted as first option, then the "suspicious values removed" as the saecond and default option, and then the adjusted. Now the first option is adjusted (they don't deal with the unadjusted any more), and the other menu options don't make much sense (they would be the same). I think that just hasn't caught up, but probably GISS should scrap the facility, and leave it to NOAA. They do it well now, and are the custodians of the data.Nick Stokeshttps://www.blogger.com/profile/06377413236983002873noreply@blogger.comtag:blogger.com,1999:blog-7729093380675162051.post-76197400579185011632012-01-31T04:06:33.758+11:002012-01-31T04:06:33.758+11:00Nick, the reason people focus on the adjusted GHCN...Nick, the reason people focus on the adjusted GHCN data, as you know, is that it's the adjusted data that is used by GISS and HADCRUT.<br /><br />On the GHCN ftp site you can get v3.0 and v3.1 from November, and see that v3.0-adjusted is, as you say, almost the same as the unadjusted.<br /><br />Can you make the v2 file you have available? Or at least plot the v2 adjusted for Reykjavik? <br /><br />The interesting question is which of the GHCN 'quality control' procedures (ha ha) are kicking in to generate these weird adjustments. Is the algorithm finding 'breakpoints' at 1940 and 1965 and 'correcting' them? Can you shed any light on this? <br /><br />Also, do you know if the GHCN v3 code is available?<br /><br />PaulMAnonymousnoreply@blogger.com