tag:blogger.com,1999:blog-7729093380675162051.post2466956322997378747..comments2021-12-18T09:28:43.436+11:00Comments on moyhu: A very different gridding method - TempLSNick Stokeshttp://www.blogger.com/profile/06377413236983002873noreply@blogger.comBlogger14125tag:blogger.com,1999:blog-7729093380675162051.post-57216974951071673072010-10-04T19:59:12.475+11:002010-10-04T19:59:12.475+11:00Kinda like being mip mapped
kool workKinda like being mip mapped<br /><br />kool workstevenhttps://www.blogger.com/profile/06920897530071011399noreply@blogger.comtag:blogger.com,1999:blog-7729093380675162051.post-40843565549231789372010-09-25T08:04:07.600+10:002010-09-25T08:04:07.600+10:00CCE,
Interesting ideas. I'm away for a couple ...CCE,<br />Interesting ideas. I'm away for a couple more days, so I'll respond more fully when I get back.Nick Stokesnoreply@blogger.comtag:blogger.com,1999:blog-7729093380675162051.post-63575375483121217122010-09-23T09:47:37.969+10:002010-09-23T09:47:37.969+10:00You've probably thought about all of this alre...You've probably thought about all of this already, but I was thinking that rather than having the algorithm create the grid, you could start with a very tight grid applied uniformly to the whole world. Find the rectangles that best encompass each area and then use your algorithm to subdivide each rectangle, (although keeping to the underlying grid). Then slice off the parts of each rectangle that overlap with other rectangles, keeping to the underlying grid and continent/ocean/zone borders.<br /><br />Some advanced compression techniques slice images into different regions, so they can be compressed according to different criteria. Is there any thing in that field that might apply to this? Your grids certainly look like compressed images, with areas of low and high resolution.ccehttps://www.blogger.com/profile/03646816472336349526noreply@blogger.comtag:blogger.com,1999:blog-7729093380675162051.post-26386395299421646672010-09-22T22:23:47.200+10:002010-09-22T22:23:47.200+10:00CCE, the problem, it seems to me, is that then the...CCE, the problem, it seems to me, is that then there would be overlapping areas. With a regular grid you can draw separators, but here with each region having ragged edges that don't fit together, you can't. I'm still not sure how best to grid regions at all. A land mask might help, but it depends on the region.<br /><br />Of course, lat-lon gridding of countries etc also has problems.Nick Stokeshttps://www.blogger.com/profile/06377413236983002873noreply@blogger.comtag:blogger.com,1999:blog-7729093380675162051.post-272547528890317232010-09-21T19:41:31.509+10:002010-09-21T19:41:31.509+10:00Re: regional analysis. I think a reasonable metho...Re: regional analysis. I think a reasonable method would be to break the world into continents and oceans, and then futher break those down by latitudinal zones. Then apply your gridding algorithm to each area. You could make manual tweaks to accomodate weird/small areas, and set the Arctic and Antarctic as their own areas regardless of continent or ocean. This should make the coverage whithin the gridboxes more uniform, and it would let you to calculate the correlation of anomalies and the uncertainty within each area seperately.ccehttps://www.blogger.com/profile/03646816472336349526noreply@blogger.comtag:blogger.com,1999:blog-7729093380675162051.post-20456649720862437272010-09-16T01:44:27.406+10:002010-09-16T01:44:27.406+10:00My own two cents: I think the best approach for t...My own two cents: I think the best approach for the uncertainty would be a Monte Carlo based approach.<br /><br />Basically you would generate a series of "realizations" of temperature which have the same time/space correlational characteristics. (E.g., high-resolution model outputs.)<br /><br />Since you know the "actual" global temperature, you can directly measure the error. Repeat this over a number of realizations, and that should allow you to assess the uncertainty associated with your method.<br /><br />The cool thing about it is you could actually test the effects of different area weightings on your net uncertainty directly.Carrickhttps://www.blogger.com/profile/03476050886656768837noreply@blogger.comtag:blogger.com,1999:blog-7729093380675162051.post-71965486989921102742010-09-16T01:33:09.856+10:002010-09-16T01:33:09.856+10:00Anyway, thanks for your comments. I had seen th...Anyway, thanks for your comments. I had seen the exchange with you and Triscari... as you correctly pointed out, his comments on prior art and irregular sampling applies only to 1-dimensional series.<br /><br />Temporal correlation in the residual certainly is a big deal and it will of course affect your estimates of the uncertainty. I have rarely used the uncertainty obtained using the fitting residuals in my estimations of uncertainty. (There are two many assumptions associated with it, stationarity of the noise being a big one.)<br /><br />In the past, with climate data, I've cheated by detrending and low-pass filtering it in e.g. 30 year segments, which removes all of the correlation. (So on 30 year periods, simple statistical methods apply.) <br /><br />I also think some noise amplification is a necessary feature of any irregular gridding (whenever you have undersampled regions compared to other more heavily sampled regions). I'll be interested in seeing what you come up with.Carrickhttps://www.blogger.com/profile/03476050886656768837noreply@blogger.comtag:blogger.com,1999:blog-7729093380675162051.post-16340822084784494642010-09-15T17:43:18.852+10:002010-09-15T17:43:18.852+10:00Carrick, it's a linear regression. If you look...Carrick, it's a linear regression. If you look at the model equations in the next post, after fitting there is a series of residuals, each of which one hopes will behave as an independent random variable, and the variance can be estimated. The variance (and covariance) of the fitted parameters is in turn estimated by applying the inverse design matrix to these variances.<br /><br />Of course, the residuals won't be independent. This has been argued about with the much simpler regression of temp trends (eg Lucia's Cochrane-Orcutt). <a href="http://rankexploits.com/musings/2010/new-work-on-temperature-reconstructions/" rel="nofollow">Here</a> is Joe Triscari (#48938 etc) taking me to task about it in this context. I've been thinking about what might be possible, but it's awkward. I'd need to fit, estimate covariance, then fit again. If I use conjugate gradient, each step has a nested CG iteration.<br /><br />I've been looking at trying to reduce the uncertainty. Both conventional gridding and my method tend to give quite variable weighting to nearby stations, and this increases variance of the weighted mean. In my pictures this shows up as fluctuating cell areas. Irregular gridding can potentially reduce this variability, by "diffusing" the weighting locally.<br /><br />Anyway, you've got me thinking about it - I'll try to post some calcs.Nick Stokeshttps://www.blogger.com/profile/06377413236983002873noreply@blogger.comtag:blogger.com,1999:blog-7729093380675162051.post-14347114294791827592010-09-15T16:37:09.405+10:002010-09-15T16:37:09.405+10:00Nick, any thoughts on how to estimate the uncertai...Nick, any thoughts on how to estimate the uncertainty with your method?Carrickhttps://www.blogger.com/profile/03476050886656768837noreply@blogger.comtag:blogger.com,1999:blog-7729093380675162051.post-33501311445632991272010-09-12T14:27:32.886+10:002010-09-12T14:27:32.886+10:00The main issue with minimum number of stations is ...The main issue with minimum number of stations is the correction needed when there is missing data. Usually I ask for 2 or 3 minimum to give a buffer before the search up the tree is needed. But in this case I also set a minimum area, so in dense regions, the number of stations was quite large.<br /><br />I've been experimenting with using a quality score rather than just number. For example, don't divide unless the stations have at least 200 (added) years of record between them.<br /><br />There was a bug in the program as displayed, in that it didn't guarantee what it was supposed to - that every cell would contain a station in every month. But the exceptions were few, and made no perceptible difference to the analysis. It's fixed now.<br /><br />The main problem with the concept is that the weighting of cells is locally rather variable. This doesn't cause bias, but does increase noise. It places the major reliance on fewer stations. Existing methods do that too, and it doesn't matter too much, because it mainly affects areas where there are stations to spare. Still, I'm trying to improve that.Nick Stokeshttps://www.blogger.com/profile/06377413236983002873noreply@blogger.comtag:blogger.com,1999:blog-7729093380675162051.post-68175474202630519732010-09-12T12:01:52.600+10:002010-09-12T12:01:52.600+10:00If I might suggest, run a series with a different ...If I might suggest, run a series with a different minimum number of stations for a grid point.EliRabetthttps://www.blogger.com/profile/07957002964638398767noreply@blogger.comtag:blogger.com,1999:blog-7729093380675162051.post-61552349300205094672010-09-04T10:27:04.011+10:002010-09-04T10:27:04.011+10:00drj, I've put all the data used for these plot...drj, I've put all the data used for these plots on mesh.txt in the zip irreg_grid.zip on the <a href="http://drop.io/erfitad" rel="nofollow"> doc store</a>. There is the complete rectangle tree, and a station dataset.Nick Stokeshttps://www.blogger.com/profile/06377413236983002873noreply@blogger.comtag:blogger.com,1999:blog-7729093380675162051.post-37345496099939517782010-09-04T07:46:30.264+10:002010-09-04T07:46:30.264+10:00drj, thanks for the comments. I'll put togethe...drj, thanks for the comments. I'll put together a file of the data that was used for these plots. Dec 2008 is just one of the months - there is a different (but related) grid for every month.Nick Stokeshttps://www.blogger.com/profile/06377413236983002873noreply@blogger.comtag:blogger.com,1999:blog-7729093380675162051.post-38386305465092273982010-09-04T02:34:03.409+10:002010-09-04T02:34:03.409+10:00Love the gridding diagrams, and the refreshingly n...Love the gridding diagrams, and the refreshingly novel approach.<br /><br />Do you have any of the gridded datafiles available? I have some code for displaying anomalies for the GISTEMP grid, and it may be fairly easy to make it general enough to display these charmingly irregular grids.Anonymousnoreply@blogger.com