Monday, March 25, 2019

Evenly spaced icosahedral grid for global temperature

I have been working on a new integration method for the next version of TempLS, V4. I call it the LOESS method; it works by local regression onto an evenly spaced set of nodes, where the anomalies can then be averaged. But the question then is, how to get that evenly spaced set of nodes.

I have written a lot about that before. The basic idea is to place points evenly on one of the five platonic solids, and then map them onto the sphere. The platonic solids are at least somewhat similar in shape to the sphere. I wrote here about using the cube (called a cubed sphere), and here with general ideas about gridding, with emphasis on the icosahedron .

A problem is that projection from the regular solid to the sphere does not preserve equal spacing. The central parts of the face are spread out by being projected further, while the areas near the vertices are actually compressed, since the face is inclined to the surface normal to the projection. For the cube, the area ratio is about 3:1; for the icosahedron it is about 2:1. In the cube post I explained a mapping that could be applied to the faces before projection, which would counter the disparity. I wrote here about how this could be applied to get an equal area projection (at the cost of cuts) for earth mapping.

I had some trouble extending this to the icosahedron, because I couldn't see a general mapping scheme with the right properties. But I now do see one based on homogeneous coordinates (also called projective or trilinear), which is simple and effective. I talk a bit about homogeneous coordinates, so I should digress to say a bit more about them. I'll put this digression in blue, so you can skip if you want.

Homogeneous coordinates

Every point x in a triangle is represented by three numbers (u1,u2,u3), which must add to 1. You can get x back, given the vertices (x1,x2,x3)
x = u1x1 + u2x2 + u3x3
and that illustrates an important role. They can interpolate properties known at the vertices by a similar formula. I'll make three points about them:

  1. The coordinates are often explained by areas. If you divided the triangle into three by connecting x to the vertices, then the proportion of area in the triangle opposite node 1 is its coordinate, etc.
  2. The coordinates are the real 3D coordinates of the equilateral triangle joining the unit points on the axes ((1,0,0), (0,1,0),(0,0,1)). The requirement to add to 1 is just the requirement that they lie on the plane through those 3 points. And since you can map those unit vertices onto the general triangle given by nodes (x1,x2,x3), you can map the interior points by multiplying by the same matrix. And it is the inverse of that matrix that takes an arbitrary triangle into the unit triangle. That is how you can calculate the homogeneous coordinates.
  3. The finite element method also has the idea of an interpolation, or basis, function. You multiply the nodal values by the values of the basis function at an interior point to get the interpolate. So homogeneous coords are just a special case of linear basis functions for one particular shape. That enables generalisation of the mapping idea I will explain.

Mapping for uniformity

A mapping before projection has some basic requirements:
  • It must keep points within the face.
  • It must comply with the symmetries of the face. The reason for this is that points on the edges will be mapped within each face that they belong to. Symmetry is needed so they do not get sent to different places.
A way of ensuring this is to apply a simple function of one variable to each projective coordinate, to make new interpolation functions. The function will have to be monotonic, and zero at zero (to keep boundary nodes on the boundary). And to comply with the requirement that the new coords add to 1, the result muct be normalised by dividing by the sum.

I chose a function of the form u/(1+a*u), because it is easily inverted (u/(1-a*u)). It is possible to determine a theoretically to make various match-ups, or just trial and error, since there is no perfect outcome. I found a value of 0.28 worked well. It means that instead of mapped triangles varying by a factor of 2, they only vary by 5%, which I think is good enough.

Here is a mapping of what is done. The icosahedron has 20 equilateral triangle faces. Each edge is divided into N equal parts. Connecting them with lines parallel to the edges makes N^2 with corresponding nodes. In this diagram, the purple triangles are the original regular grid, and the lines show the mapped grid. You'll see that the line triangles are larger near the corners, and smaller near the centre.

WebGL picture of the result

I used the WebGL facility to show the result on the sphere. In what follows, the sphere is a trackball that you can drag around. It's actually a LOESS plot of the February temperatures, which I'll say more about in a future post. But in the style of the facility, you can use the checkboxes top right to hide various things. The mesh is clearer if you clear "mesh". make 'mesh_l" go away, and you can see the points better. You can dispense with the map too, if you want.













































Tuesday, March 19, 2019

GISS February global up 0.05°C from January, third warmest February.

The GISS V3 land/ocean temperature anomaly rose 0.05°C in February. The anomaly average was 0.92°C, up from December 0.87°C. It compared with a 0.046°C rise in TempLS V3 mesh. I can't find Jim Hansen's monthly report on the web site, but his email linked to this page.

Like GISS, Moyhu is coming out with a V4 which will use GHCN V4. Actually, V3 could use it too, but it is time for a new version. GISS has marked theirs as beta - the monthly averages are here. V4 said the rise was only 0.01°C; Moyhu's V4 said the rise was higher, at 0.128°C. V4 of course has many more stations, and somewhat to my surprise, has xtensive reporting quite early; however, there is also extensive later data, and I'm still getting a feel for when it settles. My reporting is mixed; the main table shows V4, but the report and graphs are V3. This will change soon. I'll show below the anomaly maps for Moyhu and GISS V4 as well.

The overall pattern was similar to that in TempLS. W Canada and NW US were very cold, though SE US was quite warm.. Europe was warm, especially NE (Russia). Warmth also in Siberia and Alaska, and nearby Arctic. Australia was still quite warm. I'll note now that March so far is looking a whole lot warmer.

As usual here, I will compare the GISS and previous TempLS plots below the jump.

Sunday, March 10, 2019

February global surface TempLS (with GHCN V4) up 0.033°C from January.

TempLS is in transition from using GHCN V3 (which it has used for the last eight years) to the new GHCN V4. I'll report the V4 mesh results here, but for now, the Moyhu posted data is based on V3. I've been delayed a little because it is timely to also bring out a new version of TempLS (V4). Apart from anything else, the code needs reorganising to handle the bigger datasets smoothly. The Moyhu data page is still reporting values based on V3, as are the various pages with globe maps.

I worried about whether GHCN V4 would report as punctually as V3. But it seems to do better. Their version of 9 March showed 8779 stations reporting, compared with 10653 in January. But unlike V3, where countries report en bloc, here the reporting is spread, so no region is completely missing. I'll show a station map below.

TempLS mesh with V3 GHCN showed a larger rise, from 0.686°C to 0.779°C in February. This was more in line with the rise calculated with NCEP reanalysis. However, the V3 based infilled grid method was in agreement with V4 mesh, as was V4 infilled.

I'll write a post soon with a comparison of the whole V4-based history compared with V3. I'll also post a new version of TempLS which I have been using - I'll call that V4 too. It's mainly a restructuring to deal with the greater memory demands of GHCN V4, and also to include my new approach to handling spherical harmonics enhancement.

The TempLS mesh anomaly (1961-90 base) was 0.746deg;C in February vs 0.708°C in January. That makes it the third warmest February in the (V4-based) record.

The main feature of the month was rather severe cold over western Canada and adjacent US, but with warmth just a little to the North in the Arctic and Alaska. Europe was also very warm. There were patches of cool in the Sahara and western China.
Here is the temperature map.



And here is the map of land and sea stations reporting:




Monday, March 4, 2019

February NCEP/NCAR global surface anomaly up 0.097°C from January

The Moyhu NCEP/NCAR index rose from 0.296°C in January to 0.393°C in February, on a 1994-2013 anomaly base. That makes it the warmest month since May 2017. The warmth came mainly in a spurt at the end of the month, which is continuing.

It was quite cold in W Canada and the US, although warm in Alaska and the SE USA. Warm in Europe, cold in the Sahara, and mostly warm in Antarctica, with the Arctic mixed.

The BoM ENSO Outlook was is still set to Watch.





Saturday, February 16, 2019

GISS January global down 0.01°C from December, 0.11°C higher than November.

Update With this post I initially misread the GISS data file, taking values from the end of the row which are period averages, rather than the correct monthly averages for November and December, as ErikGBL noted in comments. It actually made only a small difference, but I have fixed it. I first had a small rise from December, now a small drop.
The GISS land/ocean temperature anomaly fell 0.01°C in January, which followed a 0.11°C rise the month before. The January anomaly average was 0.88°C, down from December 0.89°C. It compared with a 0.034°C fall in TempLS mesh . Jim Hansen's report is here.

The overall pattern was similar to that in TempLS. The US was actually fairly warm overall, despite the cold snap at the end of the month. Canada's Arctic islands were very cold, but a big band of warmth in NW Canada/Alaska, and another from E Siberia (which also had a cold snap) through to the Caspian region. SE Australia was hot.

As usual here, I will compare the GISS and previous TempLS plots below the jump.

Friday, February 15, 2019

Improving averaging of global temperature anomaly with fitted functions - seen as an aspect of a general method.

I have written a lot about two topics - anomalies in general, and spatial averaging of temperature anomalies. Both are headings in the Moyhu interactive index. My main point on anomalies as used in temperature statistics is that they play an essential role in reducing inhomogeneity, thus making the inevitable sample use more reliable.

But I think they have a wider use. The general pattern is that a field, temperature T, say, is partitioned:
T=E+A
where E is some kind of expected value, and A is an anomaly. That sounds vague, and there are many ways you could make such a partition. But it becomes clear when associated with the use made of it. The purpose is often to calculate some linear function of T, say L(T). Here L will be some kind of average. But suppose L is an ideal, and we actually have to rely on an approximate version L1 (eg coarse sampling).

So L(T) is approximated by L1(T)=L1(E)+L1(A). Now the objective is to choose E in such a way that
  1. L1(E) can be replaced by L(E), which is more accurate for that part
  2. L1(A) loses some of the error that marred L1(T)

I will explain with three examples here, of which the third will be developed in detail, because it bears on my long-term project of more accurately calculating surface temperatures:
  1. Temperature anomalies for global average (GISS). Here the station temperatures have some kind of mean of past values subtracted. It is usually the mean of some fixed period, eg 1961-90. Here is a typical post in which I explain why absolute temperatures are too inhomogeneous to average with the sample density we have, which the anomalies thus constructed are homogeneous enough, if only because they have approximately the same expected value (0). But in terms of this structure, they don't quite fit. The component E (climatology) that is removed is not calculated with a more accurate estimator, but is discarded, with only the anomaly average reported. But the mode of partition is still important, to be sure that the discarded part does not carry with it variation that should have been attributed to A. Using a consistent base period is one way of ensuring that.
  2. Another example is one I developed in a post at WUWT. This was about the time samples, for a single station, that go into forming a monthly average. An objection sometimes raised that characterising that by two values per day (max and min) "violates Nyquist". Nyquist doesn't really tell us about this kind of sampling, but I looked at the error introduced in sampling regularly a few times a day (eg every 12 hours). This does create aliasing of some harmonics of the diurnal frequency to zero, which adds spurious amounts to the monthly average. I showed that this is a limited error, but which can be reduced with a T=E+A partition. Now E is an estimate of the regular diurnal cycle, based on fine (eg hourly) sampling of a few reference years. That takes most of the diurnal variation out of A, and so sparse sampling (L1) aliases very little to zero, and causes little error. The fine operator (L) can be applied to the E component.
  3. The third example, to be developed here, is for the monthly spatial integration of anomalies. I'll describe it first rather generally. Suppose T is a variable to be integrated over any domain, and L1 is an approximate integration operator. Then suppose E is a Fourier fit with the first n orthogonal functions, where the inner product is integration with L1.

    Then E contains the low-frequency terms which contribute most to the integral, leaving A with just high frequency fluctuations. These may be of lower amplitude that the whole, but more importantly, they will make little contribution to the low pass integration operator. This is still true for the approximate L1, since the basic reason for the low-pass is cancellation.

    In fact, by the Fourier process, L1(A) is exactly 0, since A is orthogonal to the fitting functions, of which the lowest is a constant. So if the improved integration operator L is just exact integration, the varying fitted terms will make zero contribution to the integral, except for the constant term. The improved integral is just the integral of that constant. The improvement comes because the approximate integral of those low frequency components is replaced by the exact.

Integration methods for the sphere

I have developed four main methods for averaging temperature anomalies via numerical integration on the sphere, which I have incorporated into my monthly program TempLS, with results regularly reported here (click TempLS button). They are described in greatest detail here, with some further ideas. The methods described are
  • Simple gridding - just assign to each cell of a lat/lon grid the average of the anomalies in the cell, and then us ethe area-weighted average of the cells with data. This is basically what is done in HADCRUT
  • Infilled gridding - where cells without data are assigned a value derived from neighbouring cells in some way, and then the area-weighted average of all cells can be used. With the assignment done by kriging, this is the Cowtan and Way improvement of HADCRUT.
  • Finite element integration using an irregular triangular mesh. This is my preferred method, as it assigns to each point on Earth the interpolated value from the three nearest data points for each month.
  • Spherical Harmonics. I'll now talk more about that. In its original form, I did a least squares regression of the temperature with a set of spherical harmonics, and took the lowest coefficient as the integral.


The final method, using Spherical Harmonics (SHs), is not like the others, in that it doesn't introduce a subdivision of the area. But I now think it is best seen as not an independent method, but as an improvement applicable to any method. To just write out the regression maths, if L1(xy) is the scalar product, and H is the vector of fitting functions (SH), then the regression coefficients of the fit are

b = L1(H⊗H)-1L1(H⊗T), where ⊗ is the outer product from the weighted integration

and so b0L1(H0) is the average of T

I generally denote L1(H⊗H) as HwH, since L1 is just a weighted sum with weights w (this is the architecture of TempLS). The functions H are orthogonal wrt exact integration L, so L(H⊗H) would be a unit matrix. The approximate version deviates from this, and a figure of merit is the condition number of HwH. This is kept down if the integration method L1 is good.

In addition to the averaging methods listed above, I should add a very primitive one - the simple, unweighted average. This is generally regarded as too unreliable for use, but my original SH method, which actually performed very well, was basically the improvement of this method. You can of course expect better outcomes from improving better methods, and as I shall show, this happens. However, the relative improvement is much less. The reason is that the improvement comes from the improved integration of the SH components, and with, say, the mesh method, the finite element integration was already very good.

More on Spherical Harmonics

I described the functions here, with visualisation here, and a movie here. There are two parameters L and M; it is natural to think of M as determining the number of oscillations, with L varying from 0 to M (for fixed M) going from purely latitude oscillation to purely longitudinal. In the latter case (L=M), there are M periods of the sinusoidal variation around the equator. For SHs of order up to M, there are (M+1)2 functions, and these are the groupings that I will test below.

Tests

I looked at various means of anomalies for each of the months of 2018. I used the anomalies calculated by the mesh method; other integration schemes give slightly different values, but I don't think that changes the issues of integration. For each month and order M of SH, I calculate the mean for each of the four methods
  • No weighting (simple average)
  • Simple grid with no infill
  • Infilled grids
  • Integrating on irregular triangular mesh.
The value for M=0 is the unimproved value. Here is a plot of results. You can see the different months by using the buttons at the bottom:



You can see that the better methods, mesh and infilled grid, start out usually close together, with often the simple average furthest away. But then, as the order M incrases, they all converge for a while. Eventually this gets ragged as the higher order SH's are less well integrated by the integration scheme. The direct cause is ill-conditioning of the HwH matrix. I'll show here a similar plot of log10 of this condition number for each case.

There is an earlier discussion of SH improvements here. It shows (for Jan 2017) the sum of squares improvement, but more relevantly, a detailed map of the residuals. It shows how, after removing up to M=10, there are still spatial patterns in the residuals, some of which look like higher order harmonics.

Discussion

As you look through the months, the blue mesh line is usually fairly stable, at least up to about M=10. This indicates that the removal of SH components has little effect, because the mesh integration was already good, and replacing with exact integration makes little difference. The other lines start out separated; the black unweighted average often a long way away. But they converge toward the blue line, which says that the SH separation is giving better results.

The condition numbers also give a good quantification of the merit of the integration schemes. mesh is markedly better than infilled grid, which is in turn better than simple grid weighting, which in turn is better than no weighting at all. Better conditioning has the practical effect that mesh integration can be used to approximate temperature anomalies with a SH fit (as done monthly here) to higher order, and better resolution, than other methods.

An anomaly in approximation behaviour is February, in which there was convergence toward the blue, but that showed a gradual increase rather than stability after about M=5. I have looked into that; I don't really know what is different about the February mesh that would cause that. The meshes can be seen here. It is true that the February mesh does do a less accurate job with integrating the harmonics, as can be seen from the top row of HwH. I can't track that down to a visible difference in the mesh. The condition number for February rises earlier than it does for March, although not so different from January.

I had thought that, although subtracting fitted SHs did not improve mesh integration here, it might do so if stations were fewer. I tried that in conjunction with my culling process. However, no improvement was found there either. The reason seems to be that when stations are too sparse for SHs to be well integrated, so there is room for improvement, they are also too sparse to allow the fit to be identified. There just aren't enough degrees of freedom.

Conclusion

The idea of separating out an "expected" component prior to integration, which can then be integrated with a more accurate operator, has general application. In the case of spatial integration of monthly anomalies on the sphere, it gives a radical improvement of inferior methods. It does not do much for the better mesh integration, but the different behaviour quantifies the merits order of the various schemes.

Although the mesh method seems to be at least as good as SH upgrades of grid methods, the latter is faster if meshes would have to be calculated. Simple gridding is fast, as is the SH fitting. However, on my PC even full meshing takes only a few minutes. So I have upgraded the "SH method" in TempLS to be the enhancement of the mesh method, since I can generally re-use meshes anyway.



Thursday, February 7, 2019

January global surface TempLS down 0.033°C from December.

The TempLS mesh anomaly (1961-90 base) was 0.704deg;C in January vs 0.737°C in December. That makes it the fourth warmest January in the record.

Land temperatures were down, but SST rose a little after two months of decline. Despite the late cold snap in the mid-west, the US was not particularly cold, although the Hudson's Bay region of Canada was. The Sahara was cool too, but there were big warm patches in Siberia, the Caspian region, Alaska, and, notably, Australia.
Here is the temperature map. As always, there is a more detailed active sphere map here.