tag:blogger.com,1999:blog-7729093380675162051.post1027678748019596756..comments2021-05-08T14:09:35.707+10:00Comments on moyhu: Spatial weighting and Voronoi tessellation.Nick Stokeshttp://www.blogger.com/profile/06377413236983002873noreply@blogger.comBlogger11125tag:blogger.com,1999:blog-7729093380675162051.post-32384213337348932522011-03-06T13:55:49.869+11:002011-03-06T13:55:49.869+11:00Thanks, Nic
Yes, it wasn't hard to see.Thanks, Nic<br />Yes, it wasn't hard to see.Nick Stokeshttps://www.blogger.com/profile/06377413236983002873noreply@blogger.comtag:blogger.com,1999:blog-7729093380675162051.post-61548597748126638962011-03-06T08:49:36.844+11:002011-03-06T08:49:36.844+11:00Nick,
"That led to a corrigendum in which a Q...Nick,<br />"That led to a corrigendum in which a Quenouille correction was used. I haven't found where O10 did that, but I presume they did."<br /><br />Yes, indeed, I think we stated as much in the paper. And, unlike S09, Ryan posted full code so people can check that we actually did what we said we did. It is in lmFn, which is called by getRecon to do the trend and trend CI calculations:<br /><br />### Apply DoF correction<br />Q = sqrt((1 - r) / (1 + r))<br />SE = SE / Q<br /> <br />I think you will find that the RLS method we used in effect incorporated full spatial correlation information, as derived from the AVHRR satellite data. Although this data suffers form temporal drift and inhomogeneities, that doesn't degrade the spatial correlation information nearly as much as it affects the accuracy of its temporal data.NicLnoreply@blogger.comtag:blogger.com,1999:blog-7729093380675162051.post-3730595684116381322011-03-05T23:36:18.707+11:002011-03-05T23:36:18.707+11:00Charlie,
I think you are on to something there. It...Charlie,<br />I think you are on to something there. It's more than just spatial weighting though - it's an inadequacy in least squares. The weighting function really should incorporatee a correlation matrix. <a href="http://rankexploits.com/musings/2010/new-work-on-temperature-reconstructions/#comment-48938" rel="nofollow">Here</a> is Joe Triscari making that point.<br /><br />My defence is that much work is still done with OLS and the matrix kernel complicates the numerical algebra a lot. But yes, it should be done, one day.<br /><br />It's related to a controversy about S09 originally not allowing for first order correlation. That led to a corrigendum in which a Quenouille correction was used. I haven't found where O10 did that, but I presume they did. But this correction is a small step, and doesn't actually change the expected values - just the CI's.Nick Stokeshttps://www.blogger.com/profile/06377413236983002873noreply@blogger.comtag:blogger.com,1999:blog-7729093380675162051.post-39663101817389715372011-03-05T18:59:37.328+11:002011-03-05T18:59:37.328+11:00The smoothly weighted grid method I propose has no...The smoothly weighted grid method I propose has no trouble with missing months. <br /><br />I'm assuming that the weighting vs. distance function would stay constant over the run. I only mention the station to station correlation as a way of justifying/selecting a particular weighting function. One could also use some arbitrary weighting such as inverse distance squared. <br /><br />The handling of missing data would be no different than for traditional grid methods. In those, you just count the number of stations in the grid with data for that month and average them together. <br /><br />In terms of my proposed solution I would describe that averaging as combining all stations inside the grid cell with weighting coefficient of 1. Then of course, you divide the summed station values by the number of stations to get the average.<br /><br />In the modified method, all that changes is the weighting would not be limited to either 1 or 0. The result for each grid is simply the summation of each station value times its weight, divided by the total of all weights.<br /><br />The more I look at that method, the more I'm convinced that it is not merely a quick and dirty method, but that it is also perhaps the most accurate way of estimating values for each grid cells.<br /><br />The handling of a variety of combinations does what I would want an algorithm to do.<br /><br />For nearly co-located stations, it removes the influence of the direction between the stations. For what would be empty grid cells in a normal gridding method, the algorithm "reaches out" to include far away stations. In grids with cells in it and nearby, the far away stations are included in the calculations, but have small contributions compared to the nearby stations.<br /><br />The only other tweaking the algorithm might need is a way to truncate and ignore far away stations once a sufficient number of closer stations have been included.<br /><br />========================<br /><br />The approach I'm suggesting is so obvious that undoubtably somebody has already used it for spatial interpolation to infill missing data or something similar.<br /><br />I recognize that all this is very much a divergence away from the direction you are going and will stop cluttering up the thread at this point.Charliehttps://www.blogger.com/profile/17751567362228199326noreply@blogger.comtag:blogger.com,1999:blog-7729093380675162051.post-43669113099082291722011-03-05T17:12:01.651+11:002011-03-05T17:12:01.651+11:00Charlie,
The problem is that we mostly need a geom...Charlie,<br />The problem is that we mostly need a geometric criterion, because we have to grid monthly with the data varying. Now it's true, I expect, that the correlation function does not have to use the latest data etc.<br /><br />I'm keen to develop my diffusion idea. It's a bit like multigrid. We've got the weighting approx right on a large scale, but as you say, unsatisfactory locally. So we can then smooth locally weights without ddisturbing the wider balance.<br /><br />Specifically, I'm thinking this method. Usingthe mesh, we do a few diffusion steps (relaxation) in which, along lines of <100 km say, weight is exchanged. Each node might send out up to a third of what it has, and of course will receive back. The amount exchanged could taper with distance. Ten or so steps of that should even things up, and little weight would go beyond, say, 200 km.Nick Stokeshttps://www.blogger.com/profile/06377413236983002873noreply@blogger.comtag:blogger.com,1999:blog-7729093380675162051.post-48601376615482342562011-03-05T16:41:42.111+11:002011-03-05T16:41:42.111+11:00I see some stations, surrounded by others, that ha...I see some stations, surrounded by others, that have near zero weighting. <br /><br />We know from previous studies that stations a few km from each other will have high correlation and the correlation decays with increasing distance.<br /><br />I propose using that correlation vs distance function, combined with the distance to the grid cell being considered as the method of weighting stations.<br /><br />That avoids the underweighting of stations just because there is a nearby station located between a station and a nearby grid cell.<br /><br />In one sense, the traditional gridding of data is doing this same algorithm, but with only the discrete weighting coefficients of 0 or 1.<br /><br />I really just suggesting traditional gridding, but using a fuzzy correlation function rather than a discrete yes/no, or station in or not in the cell.<br /><br />Does this make any sense?<br /><br />My other suggestions were roughly the same, but were described using each station as a reference rather than looking at it from the point of view of the grid cell.Charliehttps://www.blogger.com/profile/17751567362228199326noreply@blogger.comtag:blogger.com,1999:blog-7729093380675162051.post-87208245941348184802011-03-05T03:10:45.874+11:002011-03-05T03:10:45.874+11:00So the bigger the area, the bigger the weighting??...So the bigger the area, the bigger the weighting??<br /><br />BTW, if you have already explained this, just point me towards the relevant post(s).<br /><br />I was drawn to this page because the graphics looked intriguing!<br /><br />:-)warmcasthttps://www.blogger.com/profile/07565080032835834724noreply@blogger.comtag:blogger.com,1999:blog-7729093380675162051.post-1083679999328704832011-03-04T23:18:48.996+11:002011-03-04T23:18:48.996+11:00Waemcast,
It's just for weighting. It doesn...Waemcast,<br />It's just for weighting. It doesn't in the end matter too much whether a node is a good representative of the exact area that's nominated. It's just accounting to ensure that regions aren't overrepresented. In that sense, if one node gets just a bit of coast, and another nearby gets a bigger chunk of interior, that's "fair". The region is weighted about right.<br /><br />However, there is some loss of information. If that situation does arise (as it does) then there should be some local averaging. My thinking is that this analysis should be done to allocate the area, and then some spatial smoothing of the weightsso that the information in "unlucky" nodes can still count.<br /><br />Quantitatively it could go like this. One could say that there is good correlation up to 250 km (say), but not so good beyond 500. So you want to have the 500 km scale properly allocated, and we've done that. The price is loss of effective dof, or increased variance, which goes with the variability of the weighting. If we can smooth the weighting locally, without spreading it beyond the 250 scale, then we ameliorate the variance issue without much loss.Nick Stokeshttps://www.blogger.com/profile/06377413236983002873noreply@blogger.comtag:blogger.com,1999:blog-7729093380675162051.post-52675079524114509642011-03-04T20:08:55.204+11:002011-03-04T20:08:55.204+11:00Looks interesting. But haven't you gone from o...Looks interesting. But haven't you gone from one extreme to another??<br /><br />eg. the coastal cells have gone from radiating out to infinity, to being restricted to the coast. Don't you need an algorithm to make sure the points/positions on the perimeter are in the centre of their cell?<br /><br />(BTW I know nothing about the methods you are using)warmcasthttps://www.blogger.com/profile/07565080032835834724noreply@blogger.comtag:blogger.com,1999:blog-7729093380675162051.post-21485120266604427472011-03-04T14:31:12.481+11:002011-03-04T14:31:12.481+11:00Sorry, Ron, I tried to give a direct like to the f...Sorry, Ron, I tried to give a direct like to the file, but maybe that doesn't work if you aren't the owner. I've replaced it with a link to the docs store (same as in the resources list, top right).<br /><br />I see I've also messed up the figs - the last two are the same. I'll fix that when I get back home.Nick Stokeshttps://www.blogger.com/profile/06377413236983002873noreply@blogger.comtag:blogger.com,1999:blog-7729093380675162051.post-11623936643880527622011-03-04T13:05:51.396+11:002011-03-04T13:05:51.396+11:00Sorry to report, I can't access the code.
But,...Sorry to report, I can't access the code.<br />But, then, some weird seems to happen when I try to log in and post as well.Ron Broberghttps://www.blogger.com/profile/00360356366869878444noreply@blogger.com