Tuesday, May 21, 2019

Higher resolution graphics for monthly surface anomaly.

Update: Following suggestions in comments, I have made a new tableau in which the new LOESS plots are compared not only with MERRA but with the ECWMF reanalysis ERA5. I think that where LOESS and MERRA disagree, ERA5 is often in the direction of LOESS. But you be the judge. There are the same 12 months of 2016 in the set.

I have for several years posted a monthly plot of global surface temperature anomalies, calculated by TempLS, and followed up a week or so later comparing it with the corresponding plot from GISS NASA. I use the same style. I chose GISS because it is, IMO, the best of the major organisation plots.

The TempLS plot uses spherical harmonics (SH) to give a smoothed stable plot to anomalies defined at scattered points. It has worked well, but the smoothing restricts to contours of a rather limited curvature, and of course restricts resolution. There is also an issue, noted in my previous post, that the spatial distribution of the anomalies calculated on a fixed interval, like GISS 1951-80, does not quite correspond to mine, which are derived on another basis (least squares fitting) and adjusted to the time interval by adding constants that make the means match. Ensuring that correspondence has some minor effects.

I have a new method of averaging using LOESS on a regular array of nodes derived from a regular icosahedron. This also makes possible a plot of the anomalies based on this local averaging which has apparently higher resolution than the SH based plot. The problem with SH is that resolution is limited by the areas of weakest coverage, while LOESS is truly local, and gives good coverage where the coverage is good, which for GHCN V4 is most of the world. I want to show that the apparent higher resolution is real by comparing with NASA'a MERRA reanalysis, which is posted to approx 1/2 ° resolution.

One might ask - why not just use MERRA? I don't present MERRA as the gold standard of accuracy. But in any case, it isn't available with the immediacy of surface data. I got the MERRA data from KNMI Explorer, where it is current up to end 2016. So my comparisons are for that period.

State of the art

The traditional style of mapping anomalies is to just show the lat/lon grid used colored by temperature. Here you see HADCRUT and NOAA:

In fact HADCRUT has been smoothed and is actually not such a bad plot, except for the white blanks. The NOAA plot (actually land only, from here) is mainly cited by contrarians to claim poor coverage. NOAA assists them here by posting a map from early in the month (Jan 11 here) and then not updating as more data comes in. But both have a similar failing. They treat the arbitrary grid segments as telling something about the data. But they would tell something different if a different grid size was chosen. You don't suddenly become ignorant about temperature because an arbitrary grid boundary has been reached. NOAA seems to realise this, because their corresponding land/ocean plot looks like this:

They have done a lot of infilling. I've never seen an explanation of the basis used, but here I mainly want to focus on resolution, which is still poor. The corresponding GISS plots come with two stated interpolation lengths:

On left is 250 km, right 1200km. The left should have higher resolution, but is again marred by grey blocks where it is deemed that information ceases at a cell boundary. The right fills more area with more smoothing, and is the one I use, though it still has some grey.

The new method, compared

Details of the new method are:
  • The anomaly data is first converted to the 1951-80 base, using the distribution described here.
  • An array of 5762 23042 regularly spaced nodes is created on the Earth's surface, based on dividing the regular icosahedron, with a transformation on each triangle to preserve equal area triangles (described here).
  • A first order LOESS (local regression) based on nearest 20 stations is used to interpolate each month's anomaly data onto these nodes (described here)
  • Using the regular icosahedral grid, those data are then interpolated onto the image of a 1° lat/lon array, projected onto the sphere
  • The data thus interpolated is then used to made a 2D lat/lon plot, using, as I usually do, the GISS temperature levels and colors.
I do this for each month of 2016, and compare with
  • The two graphs I usually post - GISS 1200 km and TempLS Spherical Harmonics, both 1951-80 base, with TempLS converted regionally
  • the new method is bottom right, and on right, the MERRA plot (data from KNMI), also converted regionally from 1981-2010 to 1951-80.
With the buttons at the bottom, you can cycle through the months of 2016. December displays first.

The LOESS bottom left plot looks to have more detail than the top plots. See, for example, the US plot. To test whether the appearance of resolution is real, compare with the adjacent MERRA plot. I'll mention these points in the December plot:
  • First the weakest point - LOESS shows a big cold spot in Alaska, which is not really in either MERRA or GISS. However, if you look at the 250km GISS plot above, there is a cold spot there too.
  • For Canada and ConUS, the extra detail of LOESS does seem to align with MERRA
  • The extra detail of the hot spot over NW China seems to line up
  • MERRA has a lot of detail about Antarctica, where GISS is very patchy. TempLS SH is sounder there, but LOESS moves a little more in the direction of MERRA
  • LOESS, like GISS and MERRA, has an El Nino jet, where SH is rather wavery.
You can check for other points here, and in the other months of 2016. But I think in almost all cases, where LOESS has extra detail, it will correspond to something in MERRA.

I plan to use the new graphics scheme for the WebGL page for past monthly anomalies. That page is currently hard to maintain because GHCN V4 keeps introducing new stations, which means the data does not match the stored meshes. LOESS will fix that, as well as being better.

Update - including ERA 5

In comments, Bryan Oz4caster and Zeke Hausfather recommended using the ECWMF reanalysis ERA5, perhaps in preference to MERRA. I certainly think it is a good idea to have another reanalysis, so that where TempLS disagrees with MERRA, which if any is wrong. So I downloaded ERA5 from KNMI, and made another tableau below. This time the two reanalyses are on the right, and I have duplicated TempLS LOESS on the left. For ERA5 I have used 0.5° grid spacing, similar to MERRA.

My general impression is that MERRA sometimes seems overly dramatic, and ERA5 is more in line with LOESS on such occasions. Looking at December, the cool in western Canada is similar between LOESS and ERA5. The blobs of warmth around Mongolia in ERA5 and LOESS are similar; MERRA a little different. But there still isn't any support for the cold spot in Alaska.


  1. Very nice work Nick !

    I am wondering how you got 5762 nodes. In my terminology it looks like you using a level 5 grid (10224 nodes) but then apply a transformation to get equal area triangles. Are you sure there are no gaps?

  2. Clive,
    I am actually wrong there. I was using finer resolution, in which each edge is divided into 48 parts, creating 23042 nodes (46080 triangles).
    I was thinking of the division I use for quadrature, which divides edges into 24. That gives 5762 nodes. Each face has 24^2=576 triangles, making N=11520 altogether. Then the rule is that there are N/2+2 nodes. The mapping doesn't change numbers.

    I've agreed with you for a long time that irregular mesh is the best method of quadrature. But I now think LOESS is a worthy rival, and the graphics possibilities are a bonus.

  3. Interesting work Nick. I still prefer the reanalysis approach in general, since it uses a coupled air/land/ocean model with a wide variety of meteorological inputs to effectively infill temperature for sparse areas, which could potentially be more accurate than simple temperature anomaly interpolations. That said, I'm not sure the longer running reanalyses that include 1951-1980 provide a very accurate baseline for reference. The data gaps in the 1950's and 1960's were much larger than since about 1980, especially over the oceans.

    I do like your old LS Mesh approach for temperature anomaly only analyses and your new LS V4 approach is intriguing for that same data set. Might be interesting to compare your new LS V4 monthly temperature anomaly global maps with those provided at the Copernicus web site produced using ERA5, like the ones for April 2019 and May 2018 to April 2019 here.

    1. I'd second the recommendation to use ERA5 (which is also on KNMI). MERRA is a tad dated at this point and suffers from some temporal homogeniety issues arising from changing data input streams (and biases in those), though this would tend to affect trend maps more than individual monthly anomalies.

    2. Thanks, Zeke and Bryan,
      I'll do that

  4. I have added a tableau incorporating both MERRA and ERA5. I've done the same period in 2016. I think ERA5 is often more in agreement with LOESS than is MERRA.

    1. Nick, thanks for taking the trouble to add the ERA5 output for comparison. Your LOESS results overall compare very well. I do note a fair number of discrepancies in the Alaska and Greenland areas, however, but I'm not sure what might be the cause, especially in Alaska. Maybe Zeke might have some ideas on these discrepancies?

    2. Bryan,
      I'm using unadjusted GHCN data. A possibility is that there are a few stations in Alaska for which this hurts badly. I'll look into it at some stage.

    3. Nick, I would not expect ERA5 to use adjusted station data, since hourly data are input, not monthly. Therefore, I would expect that your use of raw GHCN should be compatible in that regard.