Normally this might not matter much, because the missing areas are on average, average. But recently the Arctic, with much missing area, has been warming rapidly, and HADCRUT has been missing that. One C&W remedy was to use kriging interpolation. Another was to make a hybrid with UAH satellite tropospheric measures, which covers lot of the missing region. Importantly, they got similar results, with a trend that seemed to allow properly for the Arctic warming.
I wrote follow-up posts on C&W here, here and here. One observation was that latitude averaging would be better than hemispheric, since infilling with the latitude average was likely to be closer. And that gave results somewhat similar to C&W.
I'm going to develop this. But meanwhile, I want to show visually just what C&W does. I have made a WebGL active plot, which shows with shading the trends over various user-chosen time intervals. For HADCRUT, I explicitly infilled cells with the Hemisphere average. I show C&W with kriging - no infill is needed. So for HADCRUT areas with little data will show with the hemisphere average trend. With WebGL it is inonvenient to color cells as rectangles, so I have used shading. The plot and discussion are below.
Update: I have converted to showing grid cells as rectangles, which I think is clearer
Here is the plot:
You can select a dataset (currently HADCRUT 4 or C&W kriging), a start year and and end year. We're limited in years by the C&W data. Click "Plot New" when you have made a selection. As usual, the earth is a trackball which you can drag; the orient button will set it to map orientation, keeping your current center in place. You can right click and drag up/down to zoom.
For discussion, I'll show here two images for the period 2003-2012:
| HADCRUT 4 Trends 2003-2012 | C&W Kriging Trends 2003-2012 |
 |
 |
You can see in the HADCRUT plot a set of reddish strips. Those are cells with data. The remaining greenish area consists of cells that have been assigned the hemisphere average, and so reflect that lower trend. That is arbitrary, and you can see the contrast. The Cowtan and Way plot on the right has infilled with Kriging interpolation. The colors don't quite correspond, but you can see how the hemisphere infill is replaced by local values.
Update. I should add a caution here. With HADCRUT, I infilled empty cells with hemisphere averages for each month. Where there is no data over the period, you see the color of the hemisphere average. Where there is, you see the correct trend. But where there is a mix of infill and data over the period, you see a mixed trend. I think some of the yellow strips around the Pole reflect that.
If you want to compare, I suggest running two browsers side by side. Use Ctrl- to shrink to fit two on the screen.
I'll post soon on other ways of interpolating; Kriging is fine, but I think any reasonable scheme will do. TempLS can do it with linear interpolation on a regular triangular mesh.
Update: Oale in comments has sent along this difference graphic, highlighting the difference between the plots:
Posts
This is a very useful contribution on the subject which highlights the main issue with the original method used. I might suggest that the color scales being different does affect how one interprets the data. I notice that the plot is fairly smoothed - what method are you using for this?
ReplyDeleteRobert,
DeleteOn smoothing, I make a triangular mesh involving the centres of the grids as nodes. Then WebGL shades in between. Each grid centre should have the correct color for the grid.
I'd like to be able to assign a color to each grid cell, but the WebGL scheme I use makes it harder.
I could try to align the color maps. I'm using a simplifying scheme that I described here which has its own way of doing it, but I could override it.
Can you show the datapoints and the standard error map for the kriged result?
ReplyDeleteHans,
DeleteThe data is actually grid cells. It would be useful to add an optional overlay showing the centres or boundaries of the cells with data - I'll see if I can.
Yes, I could do a map with kriging se's; I'll put it on the list.
Off Topic: GISS August L&O - 70
ReplyDeleteThanks, JCH.,
DeleteThat's up from 0.53 to 0.7, so it's quite a big rise. TempLS went up 0.1, which I commented was about in line with SST.
What is amazes me is nobody I can find spends much time on why it dropped, and why it popped back up. I wish there was a way to close the budget each month. Early in the year ONI was strongly negative, and there were record warmest months. Now ONI is hovering around zero; July drops like a rock; and August sets a record for the GISS index. Based upon SST maps I was expecting a rebound, but I have a hard time believing it's all SST's. Looks like the land is finally warming up.
DeleteIf ENSO is driving it there is a time lag in the response to this forcing.
DeleteThat's an empirical statement, though, and doesn't explain the origin of the observed time lag.
Link is wrong, this is right:
Deletehttps://dl.dropboxusercontent.com/u/4520911/Climate/Oscillations/enso-corr.jpg
That's an average lag for ENSO of about 6 months, based on correlating SOI to global temperature rise
DeleteHere is an important correlation between oscillations in tidal gauges and ENSO that I discovered yesterday:
http://azimuth.mathforge.org/discussion/1480/tidal-records-and-enso/
-- using Sydney harbor gauges.
Nick, you ought to come over and contribute to the Azimuth forum, we are working the El Nino prediction project.
WHT,
DeleteI have registered for the forum. I did reply to your comment there, and another, but this went to spam. It's an Akismet problem I now have at all Wordpress sites. It dates from my suspension at WUWT. The mechanism there is that your comments are assigned to spam. AW said the suspension was for two days, but didn't lift it, so my attempts to comment went to spam, and this was reported to Akismet. Now I have great difficulty at all Wordpress sites. So I'm actually hoping that forum registration will bypass this.
Nick, your registration worked and I saw your suggestion on the forum. thanks.
DeleteSOI is a measure of pressure, so seeing a time-delay correlation between sea surface height (SSH) and westward tidal gauges sounds reasonable. Your instinct that it's driven by a deterministic phenomenon is I think a good one.
DeletePressure variations on the ocean surface act as a source for western-propogating Rossby waves. There is already literature on this. See for example this.
I agree with Nicks' suggest to correlate SSH with individual pressure gauges.
Thank you, the resultant 'difference' graphics between the two. Not sure how the program I use handles various values and the differences of scales but it makes for a pretty graphic for a blog: http://3.bp.blogspot.com/-Aq5l_OaPe_8/VBcjBrYtjaI/AAAAAAAAAag/H5shXbIazj0/s1600/hadcrutvscowtanway.png
ReplyDeleteThanks, Oale,
DeleteIt's a great plot. I've added it above - hope that's OK.
I think the black is the smallest difference, violets, reds and blues show a small difference while the mid spectrum shows larger values and sure it's ok.
Deleteb&w with the square projection, lost the outlines of the continents in process... http://2.bp.blogspot.com/-mWChDZYbqjg/VBkH-dbjCdI/AAAAAAAAAa8/okTFVV_PJD4/s1600/hadcrutcowtansquarebw.png
DeleteAgreed on the need for a correction. Of course the trick is getting the correction right and how you know that you did it correctly.
ReplyDeleteIndeed so. I think one should always start with the best try, and then try to improve and verify. Not interpolating at all is a worse option.
DeleteTechnically what HadCRUT is doing is interpolation too. They are effectively replacing cells that have no data with the global mean value.
ReplyDeleteFor the poles this will result in an underestimation of global mean trend, so some correction is certainly needed. On the other hand, you could legitimately consider this a lower bound on the correction needed.
The problem is that there isn't any data for that area, so working out what is a better choice, and being able to substantiate it, is a bit tricky.
Probably it's easier to establish bounds on the correction than it is to accurately make the correction.
I'm not sold on using satellite measurements to infill for the Arctic Ocean. I would expect it to bias the results high.
Based on the issues we've seen with BEST, I'm also not very happy with kriging. I think you'll see a smearing of land-based temperatures into the Arctic, which again would bias the results high.
I would say HadCRUT4 hybrid with MERRA or GHCN/HadSST3 hybrid with MERRA would look to be the most credible of the reconstructions they've done. (But I think all of them are an improvement over HadCRUT4).
Carrick,
Delete"Technically what HadCRUT is doing is interpolation too."
Yes, indeed, though they replace with the hemisphere mean. That's what I've done explicitly here, to emphasise the effect.
"I think you'll see a smearing of land-based temperatures into the Arctic, which again would bias the results high."
Well, that's the intent of any interpolation. It's a problem at the ice/open water interface. There might also be a query about whether sea ice was different from land.
C&W had some supplementary buoy data, some of which I think was in sea ice. I don't know the details.
Cowtan and Way have a note on their website Reconciling global temperature series, which seems to fit in with my expectations. I wish they had included UAH-hybrid though. Also, SON probably isn't a useful way to partition data for this region, but that's an aside.
DeleteAnyway, see the table on page 27.
In wintertime, when the Arctic is frozen, my conceptual model is to treat the Arctic Ocean as an extension of land. In that case, I'd expect Kriging to match up with MERRA. In the summertime, when maritime weather is in effect, I would expect Kriging to be biased high, due to the effect of smearing land temperatures into the ocean.
I'm not sure what to make of MAM, but that could be associated with an inflow of Atlantic and Pacific surface waters as the ice starts to break up (e.g., a real effect that MERRA is capturing).
Anyway, the effects aren't huge, at least for this interval. This range of values probably is a decent estimate of the systematic error in the method associated with the infilling algorithm.
Nick -- thanks for the clarification about the infill method. I need to cement that in my head.
ReplyDeleteI did a comparison of tend year trends for C&W's different algorithms.
Figure.
What is interesting to me is how well they track until the 1998 ENSO, and then there's a divergence The (centered interval) of 2006-2010 shows the most variation among methods too. Interestingly too, the MERRA reconstruction is higher than the other infill methods from circa 1998-2004. Still, I think it is probably more likely to be correct, because it has more detailed physics built into it.