Paleo plotting - new draft climate plotter

In the previous post, in response to a challenge posed by Gavin at Real Climate for better ways of visualizing Phanerozoic climate data, I showed an interactive plot mechanism which could allow the user to easily vary the time scale in a single plot environment. The mechanics really come from the Moyhu climate plotter, and I've been meaning to work up a new version of that (and it's time to update with 2013 data).

A limitation in the V2 plotter was a restriction to annual data, and it generally covered the instrumental period. This reduced the data size, which is of some importance if all the data has to be downloaded. It also facilitates post processing - eg combining or smoothing series.

In V3 I've allowed all kinds of time data. I can overcome the size p[roblem by using XMLHTTPrequest, so only requested data is downloaded. And to allow the user to easily go from a century timescale to decade (for monthly data) or million years, I need to upgrade the user controls. Also to show monthly data, I will need an automatic updating system.

I'm posting a draft version here. It doesn't yet have request download, though download times are reasonable. And a lot of the post-processing doesn't yet work. Smoothing and regresion are patchy, and the calculate facility will probably go. But the basic plotting should work, and I'd be interested in comments, especially of datasets to include.

Adjustable plotting for multiscale global temperatures

This post relates to a challenge posed by Gavin at Real Climate, to better depict the long term history of temperature. Gavin linked to datasets ranging from instrumental (150 years) to Phanærozoic (540 M yrs). I have been experimenting with user-adjustable graphs (see also gallery). The big changes of scale are a challenge, but I thought it was worth a try. The current version is rough, but I'll put in more data, with selectable axes.

The graph initially shows the last 200 years or so, but with a temperature scale big enough to cover the whole period. You can adjust the scales as follows:

• Translation - by mouse dragging in the plot space. However, zero time (before 2014) is fixed to the origin.
  
• Temp scale - by dragging vertically in the space to the left of the y-axis.
  
• Time scale - this is the critical one. You can make fine adjustments by dragging below the x-axis. But faster is to click in the region with <...>, blue, bottom left. < shrinks the scale (longer times), and the response is proportional to how far you click from the centre (about double per <).
  

The data sets given by Gavin were:

Period	Reference	Data download
0-600 Mya	Veizer et al (2000), Royer et al (2004) (updated Royer (2014))	Veizer d180, Royer04 Temp, Royer14 CO2
0-65 Mya	Zachos et al (2008), Hansen et al (2010)	Zachos/Hansen
0-5.3 Mya	Lisiecki and Raymo (2005)	LR04 Stack
0-800 kya	EPICA Dome C	Temperature Reconstruction
0-125 kya	NGRIP/Antarctic analog?	NGRIP 50yr
0-12 kya	Marcott et al (2013)	MEA12 stack (xls)
0-2 kya	Mann et al (2008), Ljungqvist (2010)	MEA08 EIV, Ljungqvist10
1880-2013 CE	GISTEMP	GISTEMP LOTI
1850-2013 CE	HadCRUT4	HadCRUT4 Global annual average, Cowtan&Way (infilled)
1850-2013 CE	Berkeley Earth	Land+Ocean annual mean

I have currently done HadCRUT4, GISTEMP LOTI, Ljungqvist10, Marcott et al (2013), EPICA Dome C,Lisiecki and Raymo (2005), Zachos and Hansen, Veizer d180. L&R is d190O data, so I scaled to to match Z&H over the first 5 million years. Veizer is ocean bottom temperature, so I used a faint color - I'm currently just trying to get the scaling to work well. So here is the plot:
Update: As Gavin at RC noted, I had L&R upside (d18O). Fixed now.

Other imperfections

I haven't been careful with anomaly base - the data is as is. I tried adjusting sequentially by overlaps, but there is noise, and for the moment, I don't think it adds value.

I have smoothed and sampled opportunistically. Epica was smoothed by three boxcar filterings, length 5, and then sampling every fifth. Zachos was smoothed once and sampled every fifth. L&R was sampled. With smoothing, end regions were included unsmoothed.

I'll surely find more.

February GISS Temp down by 0.25°C

GISS showed a substantial decrease in February, going from 0.70°C to 0.45°C. This slightly exceeded the drop shown by TempLS, which in turn was somewhat greater than the drop shown by the satellite measures. GISS had shown a rise in January, while TempLS was steady.


In agreement with TempLS, the cold places were N America and Central Asia, with warmth in NE Europe. Here is the GISS map for Feb2014:


And here, with the same scale and color scheme, is the earlier TempLS map:

Previous Months

January 2014
December
November
October
September
August
July
June
May
April
March
February
January
December 2012
November
October
September
August
July
June
May
April
March
February
January
December 2011
November
October
September
August 2011

More data and plots

TempLS global temp down 0.20°C in February

TempLS dropped substantially in February, from 0.486°C to 0.287°C. This was a little larger than the drop in the satellite indices. Again the cold spots were N America and Central Asia; warm in NE Europe.

I have made some modifications to the current data page, including a user-scalable graph.

Here is the spherical harmonics plot of the temperature distribution:



Here is the map of the 4335 stations reporting to date:

Interactive graphics gallery

A lot of new graphics has appeared here in the last year or so, so I made a new graphics gallery. It's here, or linked among the pages top right. It is itself a Javascript gadget, and shows the main graphics types, grouped by key technology (eg WebGL, Google Maps etc). You see the graphics frame at the top, and below is the explanation. The examples shown have the same format; graphic at the top, and scroll down for text. Choose a class, then one of the examples in the list below.

I won't show the gadget in this post, but below the jump is some of the detailed explanation from the page.

Graphics gallery

On this page, I have gathered graphics produced over the last three years on Moyhu. They are listed mainly by programming technique. Clicking on the "Classes" button brings up a new list of examples below. Clicking on an example shows in the frame on the right the main graphic, and below the text that was associated with it. At the bottom on the right is a link to the original post/page where it appeared.
Also at the bottom left are two buttons + and -, which allow you to expand and contract the frames. For large plots, you may want to expand the frame and unzoom the screen with Ctrl-. Ctrl+ restores.

History of methods

An important point here is that no server is used. What you see is the effect of HTML/Javascript code that you download and implement via your browser.
I use R for most arithmetic and data handling, and for static plots and images (usually png). My first advance was using Javascript for showing multiple pictures in one place, under user control. This is related to my first serious purpose here - getting more information in one place. That then bumps up against download time, so another advance was using the then new XMLHTTPRequest, which allows extra data to be downloaded on demand.
At about the same time, HTML with its 2d canvas became available. This gives JS-controlled drawing and shading. Another serious purpose was to allow proper spherical projection, with a sphere you can rotate to examine. In early times I did this with about eight perspectives with fixed images, with JS switching. But the canvas, later augmented with WebGL, gave continuous control with zooming. That's the top feature here.

The climate plotter combines the use of the canvas and JS information handling capability. Along the way, I've also experimented with the Google Maps API, mainly for met stations, and Google Earth KML files, using folders to allow large information sets.

Classes

• WebGL
  This is a browser implementation of OpenGL. It allows a Google Earth capability; you can rotate the sphere like a trackball, and zoom. Often you can click (or Shift-click) to bring up information about the closest station. The color is Gouraud shading on a triangular mesh.
  
• Animations
  These are mostly SST animations taken as screenshots from the WebGL, with possible additions.
• HTML 5 shadingThe applications here are spherical projections and data plotting. But it is very versatile.
• Active graphs
  Everything in the gallery is active. Included here are conventional graphs which can be superimposed under user control, the trend viewer, which shows a color plot of possible trends, but allows the time series to show the chosen trend lines, and the climate plotter.
• Javascript
  Everything here uses JS. I've put in this class the smaller scale visuals, with the element of user control and information density.
• Google Map apps
  I found this more useful than KML, as it gave better user control. It embeds groups of met stations in a Maps environment; you can click on the markers to pop up balloons of information. The most mabitious app is the station history, which attaches a stripped down climate plotter to allow you to plot and process station temp data.
• KML Google Earth
  If you have a GE plugin installed, this should come to life (slowly). The plug-in version has some drawbacks, the main one being that it doesn't show a folder menu, which these KML files get much of their utility from. However, it does show the stations and you can still click for info. If it doesn't work, you can click at the top. This will download the KML or KMZ file, and start up GE, if you have it installed. Then you can use the folders.
  
  There's a who;e world of GE API programming - this may get better.

McIntyre, Mann, and the Gaspe cedars

Sixteen years later, people are still arguing about Mann's 1998 Nature paper (MBH98) on multiproxy (112) reconstructions. McIntyre and McKitrick (M&M) wrote a paper in Energy and Environment in 2005, and I find myself still arguing about that.

The 2005 paper was a major resource for Congressman Barton's 2006 inquisition of Mann. One of its featured criticisms was the Gaspe dataset, number 53, and in particular the fact that Mann had padded some missing data from 1400 to 1403 with the 1404 value. And Barton directed Mann et al to explain.

One thing not often now mentioned is that in that paper, M&M actually did what many other critics should have done. They repeated the calculation with their criticism made good, to see what effect it had. This was in their Figure 1. They showed the effect of marking those four Gaspe years as missing, and then the effect of using a centered mean rather than Mann's famous calibration mean. They got a surprisingly large difference, which has been much cited in recent days. This post reports on my investigation of that surprise.

Barton got statistician Wegman to report, and his report was heavily based on the M&M 2005 papers. He said of this
"In MBH98, the first four years of both of these series were extrapolated. The extrapolation has the effect of depressing early 15th century results, and was not disclosed by Mann et al. until a later paper, "
This depressing effect was based on the M&M calculation reported in their Fig 1.

I'll show that the effect of padding those four years was limited to just those four years, and is minor. The effect Wegman describes was due to a coding rule in which M&M in effect removed the effect of Gaspe not just the four years in question, but all years from 1400-1450.

I have previously criticized the use Wegman made of the M&M 2005 GRL code for supposedly showing how noise could generate hockey sticks. There there was a clearly deliberate selection of the top 100, ordered by "hockey stick index", from which the displayed samples were chosen. In the case described here, there is no indication that the amplification of the effect was deliberate. In fact, M&M say they are following a policy of Mann's. However, the effect as presented is misleading, in that the change shown is due to the application of a rule rather than the actual effect of the padding.

M&M Fig 1, seen in perspective

Here is the original M&M05 Fig 1

The top panel shows their MBH emulation, which they say appears to be completely accurate. The second shows supposed effect of replacing the 1400-1403 years of proxy 53 (Gaspe) by NA, the R marker for missing. There is a moderate deviation in the years 1400-1450.

The bottom panel shows the effect of changing the offest mean to centered. I'll say more about that in a future post.

I'm rerunning the M&M code, so I'll show that my version does indeed reproduce that plot. I've used a bit of color:


Now I'll plot the same data but superimposed. This emphasises that the Gaspe deviation affects the years 1400-1450.

I've made the panel 3 plot faint, since it isn't the current subject.

For the remaining plots, I'll focus on the years 1400-1450. Here is again the data from panels 1 and 2

Padding with numbers

Mann used the 1404 value (0.723) to pad, and as Wegman suggests, this was a low point. The range from 1404-1450 was from 0.618 to 1.351. So I tried using 1.351, and got this:



This compares Mann's padding with the effect of padding with that max. It isn't very different at all, even using the max, and if you focus on the annual non-smoothed values, only the 1400-1403 values are changed. Much less effect than with NA.

What the M&M code does.

The recon work is done in a function NHbeta() in the file ee2005functions.r. They loop over the 11 periods, of which the last is 1400-1450. For each period they set up a "roster" of proxies that will be used. The statement defining it is:
roster<-!is.na(proxy[period[k+1],])

Here period[k+1] points to the first data point in the period, ie year 1400. So by setting the 1400 value to NA, the program removes Gaspe entirely for that period. They might as well have set the values from 1400-1449 to NA. In fact, I checked that doing that has the same effect.

Conclusion

Padding as Mann did with the Gaspe cedars had a very small, local effect. The M&M sffort to replace with missing values triggered a large response. But it was an artefact.