tag:blogger.com,1999:blog-7729093380675162051.post5121616894808807238..comments2024-03-28T13:56:47.604+11:00Comments on moyhu: What Steve McIntyre won't show you - nowNick Stokeshttp://www.blogger.com/profile/06377413236983002873noreply@blogger.comBlogger124125tag:blogger.com,1999:blog-7729093380675162051.post-24302066127992209812014-10-10T05:40:24.978+11:002014-10-10T05:40:24.978+11:00The non-warming in the 20th century is created wit...The non-warming in the 20th century is created with the noise-simulations. The divergence from proxies only makes it more probable that the proxy-trend is because of a signal, not noise.<br /><br />The HSI is an index for warmer 20th century than LIA. Outrage. It is not an index for warming in the 20th century. That is why it misses a lot. ehacnoreply@blogger.comtag:blogger.com,1999:blog-7729093380675162051.post-21241700229308485232014-10-10T04:48:57.408+11:002014-10-10T04:48:57.408+11:00I don't follow your logic.
All you've sho...I don't follow your logic.<br /><br />All you've shown is, like proxies, hockey sticks from persistent noise produced by Mann's uncentered PCA algorithm some times suffer from a 20th century divergence problem.<br /><br />That likely has consequences for the uncertainty estimation of Mann's inverse-regression method. It doesn't say anything about the utility of HSI towards describing the bias created by the short-centered PCA. <br /><br />I think we've talked in circles long enough here. Later.<br /><br />Carrickhttps://www.blogger.com/profile/03476050886656768837noreply@blogger.comtag:blogger.com,1999:blog-7729093380675162051.post-67627012293319448332014-10-10T04:38:11.399+11:002014-10-10T04:38:11.399+11:00Willard, my guess is Rule N and similar heuristics...Willard, my guess is Rule N and similar heuristics were generated back in the day when performing accurate Monte Carlos was difficult due to limits of computational power in those days. These days, just doing a brute-force validation test seems more appropriate in most cases.<br /><br />I'm hardly the world's expert on PCA/EOF/SVD/POD etc, but I would stick with cross-validation testing, with the exact details of this depending on the specifics of what we are interested in measuring and what level of error we can tolerate.Carrickhttps://www.blogger.com/profile/03476050886656768837noreply@blogger.comtag:blogger.com,1999:blog-7729093380675162051.post-7453369867411860652014-10-09T08:43:28.932+11:002014-10-09T08:43:28.932+11:00A useful metric that identifies negative trends in...A useful metric that identifies negative trends in the 20th century as hockeysticks. And of course: No warming in the 20th century is not at difference.<br /><br />That is just like the reconstructions. No warming in the 20th century.<br /><br />Must be a useful metric.ehacnoreply@blogger.comtag:blogger.com,1999:blog-7729093380675162051.post-38044392746607575822014-10-09T07:41:46.990+11:002014-10-09T07:41:46.990+11:00ehac: There are tree-ring series that diverges fr...ehac: <i> There are tree-ring series that diverges from temperature in the late 20th century. We know that. And there are tree-ring series with no divergence.<br /></i><br /><br />And we have some HSI's with positive trends and some with negative trends.<br /><br />You are trying to make a distinction without a difference.<br /><br />I think it's interesting that you can have positive HSI values (which compares means) along with negative trends. Perhaps this says something about what happens during the inverse-regresssion stage of MBH, but this fact doesn't address in any way the question of whether HSI is a useful metric or not.Carrickhttps://www.blogger.com/profile/03476050886656768837noreply@blogger.comtag:blogger.com,1999:blog-7729093380675162051.post-1489586119868723842014-10-09T07:25:58.157+11:002014-10-09T07:25:58.157+11:00I deleted the previous version of this comment (be...I deleted the previous version of this comment (because it was munged up; too badly). Now reply isn't working.<br /><br />willard---as far as I'm concerned, if you don't have an estimate of the uncertainty of a quantitative physical variable, then you don't have a valid measurement. So using significance testing works "virtually everywhere".Carrickhttps://www.blogger.com/profile/03476050886656768837noreply@blogger.comtag:blogger.com,1999:blog-7729093380675162051.post-91799016869526544742014-10-09T07:19:21.117+11:002014-10-09T07:19:21.117+11:00This comment has been removed by the author.Carrickhttps://www.blogger.com/profile/03476050886656768837noreply@blogger.comtag:blogger.com,1999:blog-7729093380675162051.post-83741208388162996002014-10-09T05:26:23.311+11:002014-10-09T05:26:23.311+11:00Carrick:
Non-selected..... Wellwell.
Did he tell...Carrick:<br /><br />Non-selected..... Wellwell.<br /><br />Did he tell you why he did not publish a non-selected portion to begin with? All oriented up. Or that was what he thought using his crappy HSI.ehacnoreply@blogger.comtag:blogger.com,1999:blog-7729093380675162051.post-23100848720989238112014-10-09T05:21:17.100+11:002014-10-09T05:21:17.100+11:00Nice logic Carrick. There are tree-ring series tha...Nice logic Carrick. There are tree-ring series that diverges from temperature in the late 20th century. We know that. And there are tree-ring series with no divergence.<br /><br />That fact will not salvage the HSI as it picks series negative trends in the 20th century as hockeysticks.ehacnoreply@blogger.comtag:blogger.com,1999:blog-7729093380675162051.post-12255663941584890912014-10-09T05:09:36.231+11:002014-10-09T05:09:36.231+11:00How does significance testing generalize to all me...How does significance testing generalize to all measurement conditions, Carrick?willardhttp://neverendingaudit.tumblr.comnoreply@blogger.comtag:blogger.com,1999:blog-7729093380675162051.post-64250024219882086112014-10-09T03:56:20.441+11:002014-10-09T03:56:20.441+11:00Pekka, very pretty analysis.
My experience with P...Pekka, very pretty analysis.<br /><br />My experience with PCA/EOF from a very different direction, so I had never been exposed to Preisendorfer's rule until I started reading climate blogs. It strikes me as a completely arbitrary rule that can’t possibly generalize to all measurement conditions. <br /><br />I've always thought the right way of truncating the series was by significance testing.Carrickhttps://www.blogger.com/profile/03476050886656768837noreply@blogger.comtag:blogger.com,1999:blog-7729093380675162051.post-3892429401980724662014-10-09T03:36:50.031+11:002014-10-09T03:36:50.031+11:00By the way here's the histogram of trends from...By the way here's the histogram of trends from M&M 2005 (100 select proxies).<br /><br />https://dl.dropboxusercontent.com/u/4520911/Climate/Proxies/MMGRL2005.hockeysticks100.slopes.jpg<br /><br />Steve McIntyre has said he will upload a larger, non-selected, portion of his simulation. If we want to study the question of how to appropriately simulate tree ring proxies, and we want to compare it to what McIntyre and McKitrick have done, this is necessary to have.Carrickhttps://www.blogger.com/profile/03476050886656768837noreply@blogger.comtag:blogger.com,1999:blog-7729093380675162051.post-29466306630710849452014-10-09T02:34:29.883+11:002014-10-09T02:34:29.883+11:00No, proxy time series are exactly a type of index....No, proxy time series are exactly a type of index. The temperature series developed from measuring tree rings is a type of index.<br /><br />Easy reality test: google "proxy index". You'll get over 800,000 matches.<br /><br />Proxies, e.g., tree rings, are a type of indicator. The indexes are the numerical values associated with indicators.<br />Carrickhttps://www.blogger.com/profile/03476050886656768837noreply@blogger.comtag:blogger.com,1999:blog-7729093380675162051.post-4911171854554795442014-10-09T02:10:52.050+11:002014-10-09T02:10:52.050+11:00Loss (attenuation) of low frequency is different t...Loss (attenuation) of low frequency is different than "flat handle".<br /><br />Nick's emulation of MBH with centered PCA doesn't have a flat handle either. Carrickhttps://www.blogger.com/profile/03476050886656768837noreply@blogger.comtag:blogger.com,1999:blog-7729093380675162051.post-21754897845345558572014-10-08T17:59:08.478+11:002014-10-08T17:59:08.478+11:00Nick,
My calculation originated from the observati...Nick,<br />My calculation originated from the observation that the high shares of explained variance of the MBH98 were based on deviations from the short-cantered mean, and thus contained large contributions from the deviation of the long period up to 1902 from this mean. As those numbers had been discussed and used in a way I consider inappropriate, I wanted to see, what the shares are, when only variability if the time series are considered.<br /><br />While the shares of variance explained are so similar in the four cases, the nature of the first PCs are not that similar. This was mentioned explicitly in WA07, I referred to them for this reason. The hockeystick is clearly visible in PC2 of cases (1) and (2), both have part of the signal also in PC1, but not at equal strength. As is well known MBH98 has the signal in PC1 and MM05 in PC4. From the variable ways hockeystick is present in different analyses we can conclude that some other variability competes at a comparable strength.<br /><br />I would consider Preisendorfer's rule more as a guide for deciding, whether more significant information can be kept by adding one more PC as indicator of the importance of the whole reminder. If Preisendorfer's rule tells to stop, but the explained variance is low, that might just tell that PCA is not a good tool in that case. It does not prove that, but adds to the suspicion. What the actual case is depends on factors that may or may not be revealed by further analysis of the case.<br /><br />Another question is the appropriates of the standard limits of Preisendorfer's rule to the decentered analysis. My hunch is that it cannot be used properly without modification. <br /><br />In this case the comparisons like that of WA07 prove that the results of MBH98 are fairly robust, but the question remains on the amount of suppression of the variability of past history that the method has robustly. It's well known that some suppression takes place, and that estimating, how strong the suppression is, is difficult - even in retrospect. It was impossible with the data available for MBH98 as far as I can see. I see the comparisons of Carrick as evidence that the suppression was substantial, but that's just a feeling from looking at the curves.<br /><br />Pekka Pirilähttps://www.blogger.com/profile/04747229036782463233noreply@blogger.comtag:blogger.com,1999:blog-7729093380675162051.post-79630769264568945082014-10-08T13:05:07.799+11:002014-10-08T13:05:07.799+11:00Thanks, William,
I've taken out the https - it...Thanks, William,<br />I've taken out the https - it seems to cause only trouble. Unfortunately, it's the default URL that Amazon sends me, so I'll have to remember.Nick Stokeshttps://www.blogger.com/profile/06377413236983002873noreply@blogger.comtag:blogger.com,1999:blog-7729093380675162051.post-59814624810538699202014-10-08T10:14:33.529+11:002014-10-08T10:14:33.529+11:00Thanks, Pekka,
As I interpret, your main point is ...Thanks, Pekka,<br />As I interpret, your main point is that it is the lack of scaling that makes the big difference with MM05 and variance calculated directly from the series.<br /><br />It seems to me there is another aspect. Rules like Preisendorfer's do a random emulation to decide where to truncate. So it also depends on how the variance is calculated in the emulation.<br /><br />I have reproduced your comment with graph at the bottom of the post.<br />Nick Stokeshttps://www.blogger.com/profile/06377413236983002873noreply@blogger.comtag:blogger.com,1999:blog-7729093380675162051.post-83696592855390718042014-10-08T09:00:42.692+11:002014-10-08T09:00:42.692+11:00As this issue has been discussed so much, I wanted...As this issue has been discussed so much, I wanted to understand it better and reproduced four variations of PCA of the NORAM1400 network:<br /><br />1) short-centered MBH98<br />2) scaling as in MBH98, but fully centered<br />3) standard scaling and centering<br />4) without scaling of the original time series (MM05)<br /><br />Otherwise the results are as reported elsewhere (e.g. in Wahl and Ammann, 2007), but I added the comparison of, how much each PCA explains by up to 10 first PCs calculating the shares from the real variability of time series used in each analysis. Thus the shares are calculated from the same total variance in cases (1) and (2), from slightly different in (3) and from substantively different in (4) where the variances of individual time series vary greatly.<br /><br />The results are shown here http://pirila.fi/energy/kuvat/NORAMexpl.png .<br /><br />What many may find surprising is that my numbers for MBH98 are very different from those shown in several places (e.g. RealClimate). The reason is that those numbers are not based on the variability of the time series, but on variances around the short-centered mean. That adds both to the total variance and to the contributions from individual PCs. In relative terms it adds much more to PC1 than to the total. Thus the resulting number does not tell about the real variability, but is very much affected by the decentering.<br /><br />I calculated my values by first orthonormalizing the basis relative to deviations from full mean (the mean affects, what's orthogonal), and by using this orthonormal basis for determining, how much the space spanned by N first PCs of the MBH98 analysis can explain.<br />Pekka Pirilähttps://www.blogger.com/profile/04747229036782463233noreply@blogger.comtag:blogger.com,1999:blog-7729093380675162051.post-27785129967342037712014-10-08T05:04:51.995+11:002014-10-08T05:04:51.995+11:00Carrick: Esper
http://upload.wikimedia.org/wikipe...Carrick: Esper<br /><br />http://upload.wikimedia.org/wikipedia/commons/5/5c/Temperature_reconstructions_0-2006_AD.jpg<br /><br />Flat handle?ehacnoreply@blogger.comtag:blogger.com,1999:blog-7729093380675162051.post-41363232746781414222014-10-08T04:45:46.379+11:002014-10-08T04:45:46.379+11:00Carrick: A proxy is not an index.
No problem ther...Carrick: A proxy is not an index.<br /><br />No problem there.<br /><br />However, that will not salvage the HSI. The one that picks series with no warming in the 20th century as hockeysticks.ehacnoreply@blogger.comtag:blogger.com,1999:blog-7729093380675162051.post-81497690976452058172014-10-08T02:19:42.568+11:002014-10-08T02:19:42.568+11:00This does sound like Cook is arguing that tree-rin...This does sound like Cook is arguing that tree-rings don't make good low frequency temperature proxies.<br /><br />The quoted statement by Cook relates to PAGES 2K, which are a series of multi-proxy reconstructions, so not related to this question.<br /><br />Carrickhttps://www.blogger.com/profile/03476050886656768837noreply@blogger.comtag:blogger.com,1999:blog-7729093380675162051.post-37603774569550604582014-10-08T02:10:49.351+11:002014-10-08T02:10:49.351+11:00ehac, will be no doubt aghast that some tree-ring ...ehac, will be no doubt aghast that some tree-ring proxies are showing a negative sign in their late-20th century trends.<br /><br />"Short version: tree-ring proxy is not a temperature index. It is invalid".<br /><br />:-P<br /><br />This goes under the saying "what's good for the goose is good for the gander."<br /><br />In case it isn't obvious, I don't accept that conclusion.Carrickhttps://www.blogger.com/profile/03476050886656768837noreply@blogger.comtag:blogger.com,1999:blog-7729093380675162051.post-2563278091625127252014-10-08T02:01:45.792+11:002014-10-08T02:01:45.792+11:00Kevin, the point I was making is, if you have a sy...Kevin, the point I was making is, if you have a system with a saturating (compressive) nonlinearity and you input a 1/f-type signal, the low-frequency portion of the system response will be more strongly attenuated than the high-frequency ones. This is easy to confirm numerically.<br /><br />I haven't tried modeling the effect of offset drift, but again, in analogy to GPS/INS systems, if you have higher fidelity but lower-frequency resolution non-tree-ring proxies to anchor your tree rings to and If the tree rings are temperature proxies (direct or indirect), then the composite reconstruction will be improve the accuracy of the reconstruction over using tree-rings by themselves.<br /><br />The divergence problem seems to be at least partly associated with preserving high-frequency coherence of the proxies in the reconstruction, than an issue with the proxies themselves.<br /><br />If I plot the location of Mann 2008 tree-ring proxies using a red marble for positive trend and a blue marble for negative trend for 1959-1998, I get this pattern:<br /><br />https://dl.dropboxusercontent.com/u/4520911/Climate/Proxies/mann08_marble.pdf<br /><br />Here's the histogram:<br /><br />https://dl.dropboxusercontent.com/u/4520911/Climate/Proxies/Mann_Trends_Hist.pdf<br />Carrickhttps://www.blogger.com/profile/03476050886656768837noreply@blogger.comtag:blogger.com,1999:blog-7729093380675162051.post-31554341213116040872014-10-07T20:07:40.910+11:002014-10-07T20:07:40.910+11:00Carrick - I'm not sure 'assumption' is...Carrick - I'm not sure 'assumption' is the proper term. Calibration and cross-validation are explicit methods for determining whether proxies are empirically correlated to the variable of interest. Tree ring widths (or density) in and of themselves tell us very little. It is the calibration and cross-validation that allow us to use these as thermometers and/or precipitation gauges. Any tree ring dataset may be correlated to temperature or precipitation, both, or neither. <br /><br />I'm not sure your belief that the relationship is more true for short temporal periods versus longer temporal periods is true. The 'divergence' problem shows that we can see higher correlations over longer timeframes than for shorter ones. I.e., the correlation is better over the past 200 years than over the past 50 years. <br /><br />Kevin O'Neillhttps://www.blogger.com/profile/06692943768484857724noreply@blogger.comtag:blogger.com,1999:blog-7729093380675162051.post-19458471838067234152014-10-07T18:37:10.330+11:002014-10-07T18:37:10.330+11:00Earth’s Current Warmth Not Seen in the Last 1,400 ...<a href="https://www.ldeo.columbia.edu/news-events/earth%E2%80%99s-current-warmth-not-seen-last-1400-years-or-more-says-study" rel="nofollow">Earth’s Current Warmth Not Seen in the Last 1,400 Years or More, Says Study</a><br /><br /><i>“This paper tells us what we already knew, except in a better, more comprehensive fashion,” said <b>study co-author Edward Cook</b>, a tree-ring scientist at Lamont-Doherty Earth Observatory who led the Asia reconstruction.</i><br /><br />Kevin O'Neillhttps://www.blogger.com/profile/06692943768484857724noreply@blogger.com