Moyhu data updates

For a while now, I have been maintaining updated data tables and graphics. Many are collected in the latest data page, but there are also the trend viewer, GHCN stations monthly map, and the hi-res NOAA SST (with updated movies). These mostly just grew, with a patchwork of daily and weekly updates.

I'm now trying to be more systematic, and to test for new data every hour. My scheme downloads only if there is new data, and consumes few resources otherwise. My hope is that all data will be processed and visible within an hour of its appearance.

I have upgraded the log at the bottom of the latest data page. This is supposed to record new data on arrival, with some diagnostics. Column 1 is date, which is actually the date listed at origin, translated to Melbourne time. The column headed "Delay" is the difference between this date and the date when processing is finished and the result should be on the website. I'm using this to find bugs. The date in the first column isn't totally reliable; it is the outcome of various systems, and may predate the actual availability of the data on the web. The second column is the name with link to the actual data. For the bigger files (size, col 3) a dialog box will ask whether to download. The "Time taken" is the time used by my computer in processing (again, for my diagnostics). Where several datasets are processed in the same hourly batch, this time is written against each of them. Currently, only the top few most recent lines of the log are useful, but new data should be correctly recorded in future.

NOAA temperature is a special case. It doesn't have the files I use in a NOAA ftp directory, but serves them with the current time attached. I have to use roundabout methods to decide whether they are new and need to be downloaded (I use their RSS file). By default they show as new every hour - I have measures to correct this, but they may not be perfect. Anyway, the times in the log for NOAA are not meaningful.

I have a scheme for doing the hourly listening only when an update is likely (assuming approx periodicity). If data arrives unexpectedly, it will be caught in a nightly processing.

It is still a bit experimental - I can't conveniently test some aspects other than just waiting for new (monthly) data to appear and be processed. But I think the basics are working.

Record warm periods

In the previous post, I cited the NOAA June temperatures report. This noted various periods of time with average temperatures that exceeded anything comparable in the record, particularly recent twelve month periods.

Sou has been tracking the average so far in this calendar year. Steve Bloom doesn't like non-physical calendar periods, and prefers the running average. I think Sou's analysis makes sense. It is the best guide to the 2015 year average, since it uses data from that year only. One could say people shouldn't focus on arbitrary year divisions, but they do.

However, if you want running twelve month averages, they are available at the maintained active plot. Just click the buttons on the side table headed Sm. twelve month running is the default (and only) smooth. Here is an example:



However, in this context, I'm a fan of polar, or radial, plots. Here curves track like a clock with time, with radial distance indicating temperature or other plot variable. I use it for ice extent here. Then there is a natural period of a year, but that isn't essential. The point is that by rolling it up, a long stretch of time can be covered with good resolution (but crowding).

So below the fold I'll show radial plots (with decade winding period) for the main surface indices, from 1950 to now. They will show clearly how warming has continued, as the curve spirals outward, even during the "hiatus". Now they are at record radius, and that is very likely to increase, at least for a little while. The reason is that the change in moving average depends on both the new readings, and the old readings that they replace. Mid-2014 was a relatively cool period, so as long as 2015 is warmer, the running mean will increase.

In these plots, there is a continuous curve in rainbow colors, violet in 1950, and ending with red now. It goes clockwise, 36°/year, crossing the y axis every decade as marked. Radial distance represents temperature anomaly (according to the source) as marked on the x axis, and shown with faint circles. The running mean is centered, so the latest at the bottom shows as Dec 2014.

So here is the NOAA Land/Ocean plot. A curiosity here is that the NOAA report says the maximum 12-month average is 0.83°C; I get 0.81, on their data. But the pattern is the same. Spiralling outward indicates warming, and it is clearly happening, with minor variations, since about the third circuit (70's). And it is now at record level (and has been for six months), and heading outward.



Here is GISS Land/Ocean, which looks much the same:



And here is HADCRUT 4. It probably shows the least adherence to uniform growth, and has more pronounced excursions in 1998, 2005, 2007 and 2010. It's still well in front now, though.



2015 will be a record year if the curve stays outside to previous levels for six more months. And as I said above, a month or two more of expansion is very likely, just on the basis of cooler mid-year 2014.

ps Arctic ice had been holding up fairly well. But it is showing signs of downturn in the last five days, dropping below 2010, and close to 2014.

NOAA up 0.02°C in June; GISS now up 0.04°C

I don't normally post separately for the NOAA NCEI global temperature anomaly index, but this month is the first where everyone is using the new V4 ERSST. There is also a revision of GISS post-V4 (thanks to GISS for the acknowledgement). So it is an opportunity too to review how well TempLS matches in the new environment.

The NOAA report has June at 0.88°C relative to 20Cen; up from 0.86°C in May. GISS is now 0.8°C, up from 0.76°C. Both these are the hottest June in the respective records, and would be close to the hottest month ever, if it were not for Feb-Mar of this year. I have described in my GISS post where it was hot and cold, with more quantitative information here; the NOAA report presents a similar account in more detail.

I have noted earlier how, as expected, TempLS mesh and GISS tend to go together, being interpolated, and NOAA and TempLS grid also have an affinity. Indeed, at times, TempLS and NOAA have been eerily close. With V4 that has been somewhat interrupted, although the general pattern still holds, and the correspondence is currently close. I'll show plots below the fold.

Incidentally, the revised GISS now shows nothing unexpected in the plot of recent monthly differences (earlier version here). Here how it looks now:



So here is the plot over 4 years of NOAA vs TempLS grid. The plots are made using this gadget, and are set to a common anomaly base of 1981-2010. The divergence earlier this year is the largest. Otherwise the correspondence is good.



And here is the comparison of TempLS mesh with GISS. I think the correspondence is better than before the ERSST change.



And here is the combined plot. I've used reddish for grid, bluish for interpolated. You can see the pairing, although they are all fairly close. The pairs come together at the end; this is just how it happened.

A breakdown of monthly temperature variation

A by-product of the new TempLS mesh is the analysis that can be done using the arrays of residuals and weights. The weights are mesh areas assigned to stations, and the global anomaly is the thus-weighted sum of residuals. This sum can be partitioned into regions. Two calculations are then of interest:
AveT = sum_R(w*r)/sum_R(w)
which is the average T for region R, and
Contrib_r = sum_R(w*r)/sum_G(w)
which is the proportional contribution R makes to the global T. That means you can see how those contributions add. So instead of saying it was cold in N America, you can see just how much that cold pulled down the global average.

Below the fold I'll show both of those for continent-sized regions, and also the ocean, for the months of 2015. Note that they are not exactly sums over the continent areas, but over the areas that are assigned to nodes within the regions. Each month has a different mesh, so these area assignations vary slightly - I'll show a third plot to quantify that. Of course, the regions don't change (except for ocean freezing), so to the extent it otherwise varies, it's an error.

I'll show how you can see what exactly contributed to the ups and downs of temperature this year.

The regions are Afr=Africa, Asia (without Siberia), Sib=Siberia (Asiatic Russia, code 222), S America, N America, AusNZ=Oceania, Europe, Ant=Antarctica, Arc=Arctic>65°N, Sea=SST, and All=Global. I follow the classification of the GHCN country codes, for which 100-199 is Africa etc. Arctic stations are removed from N Am and Sib.

Here is the plot of monthly regional averages. Land regions are very variable compared to Sea and Global. But they are mostly minor components of the sum, and the next plot will show that in perspective. Features are the famous N American cold of February, recent cold in Antarctica, and Feb/Mar warmth in Asia/Siberia. You can also see the Arctic warmth in March/April, which led to early melting.



Now here is the version weighted as contributions to the main sum, which is more informative. The black is the sum of the colors.



Firstly, SST has been gradually increasing, which pushes the total up significantly, without showing month-month fluctuations. Then you see the sharp cold spell in February N America. It has a small reducing effect, almost countered by a rise in Africa, and dwarfed by the warmth in Asia and Siberia. Consequently, Feb globally was a lot warmer than Jan, despite that cold much remarked on blogs. In May/Jun and even Apr, the cold in Antarctica had an effect, which, with the subsiding warmth in Asia/Siberia, is responsible for later cooling.

Finally, I'll show the variation in weights. There isn't much; I've shown 10x bars faintly over the land regions to make the effect more visible.



The exception is Antarctica, where the sea ice has been growing. Since that is then excluded from the SST component, land stations carry greater weight as the ocean recedes. The Arctic behaves irregularly. This reflects the fact that the mesh there is sparse, and has a degree of randomness in its response. It reflects the limits of this measure. Still, it's a minor effect overall.

I'll probably incorporate this analysis in the ongoing latest data.

GISS (new version) up by 0.03°C in June

The anomaly for GISS in June was 0.76°C (h/t JCH). This was just as I had expected, since TempLS mesh rose from 0.618°C to (now) 0.673°C. However, meanwhile the GISS May number increased from 0.71°C to 0.73°C, so the rise was not quite as great.

Behind that lies a story. As Olof noted, GISS this month switched from using ERSST V3b to ERSST v4 (I switched last month). They show some plots of the differences here. Here is the difference plot



In the new file April 2015 is the same as before, and May only increased by 0.02°C. But the plot shows an annual difference of more than 0.05°C in recent years. I plotted the monthly differences (GISS v4 - 3b) over the last five years:



April and May do look quite different. I wonder if GISS is still using v3b there? Anyway, the usual maps and commentary are below the fold:

Here is the GISS map for June 2015:


And here is the corresponding TempLS mesh map, based on spherical harmonics:




TempLS is somewhat smoothed so doesn't show the peaks of GISS. But the features are just the same. Heat in central and W Russia, and in NW N America. Cool in Antarctica, and N Atlantic, extending over Scandinavia.

You may have noticed that an error had got into my TempLS map plots recently. They were shown with mean subtracted, and so much colder. This has been fixed.

TempLS shows June anomaly up by 0.05°C

Most of the GHCN V3 data is now in, and ERSST v4, and TempLS mesh (the new V3)  shows a rise from 0.618 to 0.665°C, relative to a 1961-90 base period. TempLS grid rose from 0.674 to 0.711 °C. Usually the mesh weighted version is more likely to agree with GISS and the grid version with NOAA, though this time they both are similar.

The rise is somewhat at variance with the NCEP/NCAR index, which suggested a fairly cool month, although that is muddied somewhat by the May oddity when that index suggested more warming than TempLS mesh and GISS showed. Anyway the TempLS calc is certainly warm, and for TempLS grid, seems to be almost a record, falling just short of the 0.718°C for Feb 1998.

The warm places were western US and central Russia, with cold in Antarctica and the E Mediterranean, pretty much as indicated in the NCEP/NCAR report. As I mentioned in my previous post, you can get more detail in the regular WebGL map here, which shows the actual station anomalies with shading between.

There was a curious delay with the Canada data which slightly set back this report. The initial posted data was a duplicate (mostly) of May, and flagged as such, so TempLS rejected it. That seems to have been corrected, and now 4266 stations (incl SST cells) have reported, which is almost all that we can expect.

In other news, Arctic ice has held up well in the last month, but the last two days of JAXA show rising melt, and Neven says more heat is on the way.

Maintained WebGL map of past GHCN/SST station anomaliess

I have a maintained page here which shows anomalies for individual land stations and SST cells, averaged over each month. Land data comes from GHCN V3 unadjusted, and SST from ERSST, formerly ver 3b. It is a shaded color map, with true color for each station, and linear interpolated color in between. I think it's the best way of seeing the raw data.

Now that I am doing daily runs of TempLS mesh, and this produces much of the necessary data, I have decided to make production of these maps a subsidiary process. This saves duplication, and keeps up to date with versions (using now ERSST 4). But what may be more useful is the daily updating. That means that the latest month can be tracked progressively as data comes in. So today, I am waiting on data from Canada and some Latin American countries before posting the TempLS result for June. But you can see it here, and it shows how TempLS calculates its interim result with interpolation. Of course, you could wait till tomorrow when Canada will probably be in.

I have also extended the range back to 1900. You can examine any month in detail. I haven't yet done the annual and decade averages back beyond 1990, but soon.