tag:blogger.com,1999:blog-7729093380675162051.post1011102749937955409..comments2023-01-30T19:49:00.616+11:00Comments on moyhu: Surface temperature sparsity error modesNick Stokeshttp://www.blogger.com/profile/06377413236983002873noreply@blogger.comBlogger9125tag:blogger.com,1999:blog-7729093380675162051.post-37517275304297161382017-09-01T08:12:11.568+10:002017-09-01T08:12:11.568+10:00Thanks for this explanation. So It seems to me you...Thanks for this explanation. So It seems to me your method is not so different that Giss. Looks like interpolation with triangles instead of boxes and more refined calculations. But while boxes are not designed to fit GHCN stations, triangles follow their path (the three known nodes). I've got the feeling that may change the final result a little bit even if I don't really understand how.<br />Your next post comparing the various methods quantitatively will surely be interesting anyway.Pepnoreply@blogger.comtag:blogger.com,1999:blog-7729093380675162051.post-47771741332128197862017-08-31T19:29:31.809+10:002017-08-31T19:29:31.809+10:00Pep,
I tried to cover this in this recent post. I&...Pep,<br />I tried to cover this in <a href="https://moyhu.blogspot.com.au/2017/08/temperature-averaging-and-integrtaion.html" rel="nofollow">this recent post</a>. I'm working on a post comparing the various methods quantitatively - there is a previous effort <a href="https://moyhu.blogspot.com.au/2015/10/new-integration-methods-for-global.html" rel="nofollow">here</a>. The task of global averaging is to make a best approximation for every point that you don't have measurements for, and integrate it. With gridding, you approximate unmeasured points within a cell by the average of points that you have measured in that cell, which is fine. But if you just leave out the cells without data, you are effectively approximating them by the global average, which is not fine. They should also be approximated by some average of local cells. It's best done as a two step process - infer an average for the cell, and then atribute that to the unmeasured points. It doesn't really matter in detail how you do that, as long as the average is local. Kriging is used in mining when you really want to know a lot about a particular point - where you want to drill - and so it is worth a lot of work to optimise. But with integrating, there is a huge number of points to add up, and you don't have to work so hard to get the minimum variance estimate. Not that kriging makes a huge difference there anyway.<br /><br />On this basis, I like triangular mesh because it estimates each unmeasured value by three known nodes that are among the nearest (the Delaunay property helps ensure that is true). There is a penalty for sparse data leading to large triangles, but that would hurt any method. It also gives a continuous approximation, while a grid does not. That means that you aren't giving very different estimates as you cross arbitrary lines. The estimate could be stabilized by taking into account more points but, as said, this doesn't matter when you are summing large numbers of them.<br />Nick Stokeshttps://www.blogger.com/profile/06377413236983002873noreply@blogger.comtag:blogger.com,1999:blog-7729093380675162051.post-73312508822250435582017-08-31T17:01:27.127+10:002017-08-31T17:01:27.127+10:00Thanks ! Empty cells should be looked closer. For ...Thanks ! Empty cells should be looked closer. For the layman that's not obvious at all. When even scientists struggle to assess methods employed to infill empty cells (interpolation, kriging). An explanation of your method for empty cells compared with GISS and BEST would be welcome, even if I believe you already make a lot of effort discribing your work. An "Interpolation for dummies" explanation is required to better appreciate the discussion here.<br />In this context, that would be nice if you could explain with simple words why you think irregular triangular mesh is the best method. Pepnoreply@blogger.comtag:blogger.com,1999:blog-7729093380675162051.post-1593248887321931002017-08-30T14:19:03.617+10:002017-08-30T14:19:03.617+10:00Pep,
"Nick, you may have already explained th...Pep,<br /><i>"Nick, you may have already explained this in a precedent post but I was wondering which method you consider as the most accurate to give a global mean temperature."</i><br />Well, I still think irregular triangular mesh is best. I look for other methods partly to seek agreement, and also because, as with these modes, they can tell you something else. I don't think there is really much difference, though, provided you do something about empty cells. GISS interpolates, which should work. I think kriging is fine, but overkill. Every point which isn't measured is, or should be, estimated from local values. There is sufficient noise that looking for perfect interpolation isn't really hlping much. I did a comparison of methods <a href="https://moyhu.blogspot.com.au/2015/10/new-integration-methods-for-global.html" rel="nofollow">here</a>. It's one approach to a quality standard.<br /><br />As for BEST, I think what you have in mind is their use of least squares to avoid requiring a fixed anomaly base interval, which tends to exclude stations which don't have ata there. I have used that (pre-BEST) in TempLS too, and I think it is the right thing to do.<br />Nick Stokeshttps://www.blogger.com/profile/06377413236983002873noreply@blogger.comtag:blogger.com,1999:blog-7729093380675162051.post-40409562030460083272017-08-29T07:59:23.702+10:002017-08-29T07:59:23.702+10:00Actually the only way that one can evaluate whethe...Actually the only way that one can evaluate whether an integration works on an extrapolated interval is by using a physical model. Otherwise, it is basic interpolation using knowledge of the neighboring points. If there are enough next-near neighbors one might be able to infer some sort of sinusoidal signal, but again that would only work effectively for something as obvious as the ENSO dipole. There might be some other dipoles, such as the Arctic Oscillation or North Atlantic Oscillation that you could asses whether it would work in the "Real World"<br /><br />ENSO is a partial differential equation and being able to separate the spatial from the temporal allows one to do two independent integrations. Remember that the temporal oscillation will screw up whatever you do for the spatial extrapolation unless you take that into account, i.e predicting the sign of the excursion at any one point in time. And remember that ENSO has a HUGE and SIGNIFICANT effect on global temperature at any point in time.<br /><br />I am operating in the real world with all the physics that I can apply.<br />@whuthttps://www.blogger.com/profile/18297101284358849575noreply@blogger.comtag:blogger.com,1999:blog-7729093380675162051.post-67290078856732086912017-08-29T07:15:26.538+10:002017-08-29T07:15:26.538+10:00Thanks. Maybe you're talking more about the di...Thanks. Maybe you're talking more about the difficulty of climate models to predict El Nino. I was rather considering global temperature datasets and different methods to give a global estimation of the world temperature.<br />Nick, you may have already explained this in a precedent post but I was wondering which method you consider as the most accurate to give a global mean temperature.<br />As you do yourself integration of GHCN, you probably have an opinion about Gistemp, Hadcrut, NOAA, JMA and Berkeley. Statistical methods have probably an influence on the final result. I read kriging was one of the most accurate interpolation technique but I'm far from being a specialist, that's why I'm asking.<br />Gistemp has also an interpolation method, though it may be more related to neighboring stations rather than statistical correlations as Cowtan's kriging ? Giss also uses satellites to improve it's data.<br />As for Berkeley, it seems they do also statistical treatment of data to incorporate more stations. Anyway, I may be wrong with what I said but I was wondering if there is a way to assess all those methods.I believe you try to make your TempLS as accurate as possible and you may have something like a quality standard in mind.Pepnoreply@blogger.comtag:blogger.com,1999:blog-7729093380675162051.post-53244500225698858612017-08-29T03:44:56.937+10:002017-08-29T03:44:56.937+10:00"What I'd like to know is if there is à w...<i>"What I'd like to know is if there is à way to assess if integration is done correctly. In general, is there à way to assess how à dataset is able to reflect the Real World ?"</i><br /><br />The way to do this is to run <a href="https://en.wikipedia.org/wiki/Cross-validation_(statistics)" rel="nofollow">cross-validation</a> tests. If you have a physics-based model, this works very well because you can test if your integration extrapolates from the training interval over to the test interval.<br /><br />For a standing wave phenomenon such as ENSO, there are two aspects, a spatial one and a temporal one. The spatial aspect is super easy to cross-validate, as the ENSO forms an almost perfect spatial dipole that shows opposite signs at Darwin and Tahiti. So that at any one time, one can show that if you have knowledge of the temperature or atmospheric pressure at Darwin, you can accurately predict the temperature or pressure at Tahiti by reversing the value of the anomaly.<br /><br />For ENSO temporal cross-validation, this has been a longstanding challenge and a problem that no one has been able to solve. But I can demonstrate it convincingly here by assuming the lunar tidal forcing:<br /><br /><a href="http://contextearth.com/2017/08/08/enso-split-training-for-cross-validation/" rel="nofollow">http://contextearth.com/2017/08/08/enso-split-training-for-cross-validation/</a><br /><br />With this lunisolar model of ENSO, one can take any time interval of the ENSO measure and predict the value at any point in time backward or forward.<br /><br />@whuthttps://www.blogger.com/profile/18297101284358849575noreply@blogger.comtag:blogger.com,1999:blog-7729093380675162051.post-91029343998403539522017-08-29T00:36:21.731+10:002017-08-29T00:36:21.731+10:00What I'd like to know is if there is à way to ...What I'd like to know is if there is à way to assess if integration is done correctly. In general, is there à way to assess how à dataset is able to reflect the Real World ?Pepnoreply@blogger.comtag:blogger.com,1999:blog-7729093380675162051.post-21547014618039552032017-08-26T06:00:18.191+10:002017-08-26T06:00:18.191+10:00Talking about spatio-temporal relationships, here ...Talking about spatio-temporal relationships, here is a lone ranger research effort that demonstrates how much we can still discover with respect to geophysics<br /><br /><a href="http://contextearth.com/2017/08/24/lunisolar-forcing-of-earthquakes/" rel="nofollow">ContextEarth.com/2017/08/24/lunisolar-forcing-of-earthquakes</a><br /><br />Had some discussion on whether he somehow mistakenly or accidentally plotted X = X, but I don't think so.<br /><br />@whuthttps://www.blogger.com/profile/18297101284358849575noreply@blogger.com