tag:blogger.com,1999:blog-7729093380675162051.post4980164930438959248..comments2021-07-24T07:16:06.814+10:00Comments on moyhu: Trends, breakpoints and derivativesNick Stokeshttp://www.blogger.com/profile/06377413236983002873noreply@blogger.comBlogger8125tag:blogger.com,1999:blog-7729093380675162051.post-31635794285701182382015-03-04T19:00:58.615+11:002015-03-04T19:00:58.615+11:00Oh, no, it's my error. You were applying W to ...Oh, no, it's my error. You were applying W to dT/dt not to TS Care needs to be taken with shifting the kernel up or down since this will be adding a constant to each element and hence adding a fixed dT/dt to the whole series. Gregnoreply@blogger.comtag:blogger.com,1999:blog-7729093380675162051.post-77048930294274851812015-03-04T18:39:15.710+11:002015-03-04T18:39:15.710+11:00Consider the following: running mean of diff is s...Consider the following: running mean of diff is same as diff of RM. The kernel that achieves that has just two non zero values of opposite sign, one at each end of the kernel. Its the two point diff of the rectangular RM kernel.<br /><br />( This is in fact the same thing as the pairs of points Pekka referred to in the other thread, I'll come back to that ). <br /><br />As you have pointed out the OLS regression slope is the correlation of the data with a kernel of unit slope. If that is centred to be zero mean, the diff of that kernel is a positive rectangular window with a negative point as first and last element. These points are not unimportant and it cannot be shifted up to make it a simple rectangle. <br /><br />Your Welch window applied to TS is the same as ramp applied to dT/dt and so corresponds to the <b>OLS trend in dT/dt </b> ie the acceleration trend that you started out discussing, not the OLS trend of the TS. Again, this cannot be arbitrarily shifted up or down. <br /><br />Your integral maths was good but there is a logical error in how you interpret the terms. <br /><br /><br />Gregnoreply@blogger.comtag:blogger.com,1999:blog-7729093380675162051.post-47406574173346334812015-03-04T08:50:11.521+11:002015-03-04T08:50:11.521+11:00"Isn't this the same as saying that OLS t...<i>"Isn't this the same as saying that OLS trend is identical to mean of dT/dt ?"</i><br />No. As said in the recent post, trend is the Welch-weighted mean of dT/dt.Nick Stokeshttps://www.blogger.com/profile/06377413236983002873noreply@blogger.comtag:blogger.com,1999:blog-7729093380675162051.post-36615083766917060192015-03-04T05:48:54.006+11:002015-03-04T05:48:54.006+11:00"A useful property to remember about the ordi..."A useful property to remember about the ordinary mean is that it is the number which, when subtracted, minimises the sum of squares. There is a corresponding property for OLS trend. "<br /><br />Isn't this the same as saying that OLS trend is identical to mean of dT/dt ?<br />Greg Goodmannoreply@blogger.comtag:blogger.com,1999:blog-7729093380675162051.post-13222859304191739242015-02-21T19:12:14.105+11:002015-02-21T19:12:14.105+11:00Everett, the definition of an ideal low-pass filt...Everett, the definition of an ideal low-pass filter is step function in the frequency response. However, this requires a theoretically infinitely long sinc fn kernel to achieve. The rest of the game is how to approach that idea in a calculable way. <br /><br />In electronics and audio, it is usual to have a continuous stream of data and so IIR solutions are often used. In image processing, as in time series analysis, kernel based FIR solutions are more appropriate.<br /><br />The question is then the best way to window the sinc fn to make it finite without causing too much distortion. <br /><br />The Lanczos filter is one of the best. It has a fast transition band and low ripple in stop- and pass-bands.<br /><br />http://climategrog.wordpress.com/2013/11/28/lanczos-filter-script/<br /><br />It has the disadvantage of requiring a rather longer kernel than other options but is a very good filter.<br /><br />Greg Goodmanhttps://climategrog.wordpress.com/2013/11/28/lanczos-filter-script/noreply@blogger.comtag:blogger.com,1999:blog-7729093380675162051.post-88877825071774447412015-01-26T08:20:38.515+11:002015-01-26T08:20:38.515+11:00I think I mean the re-trended PDO, which looks pre...I think I mean the re-trended PDO, which looks pretty much like the SAT until around 1980 to 1985, at which time it makes a prolonged excursion in another direction.JCHnoreply@blogger.comtag:blogger.com,1999:blog-7729093380675162051.post-61032626101922696572015-01-22T07:52:22.959+11:002015-01-22T07:52:22.959+11:00Nick I should mention that the problem for derivat...Nick I should mention that the problem for derivative filters is slightly easier with climate (and many related signal types) than the general problem, because the spectrum already has a 1/f^nu, nu > 1, character to it. <br /><br />Like you, I don't stick with a rectangular window for general data (I typically find rectangular windows work well enough for temperature data usually). Instead, I use window functions that include Welch, Hann, Blackman and modified Gaussian (this is a tunable window, which has advantages).<br /><br />Carrickhttps://www.blogger.com/profile/03476050886656768837noreply@blogger.comtag:blogger.com,1999:blog-7729093380675162051.post-18856241425374077842015-01-21T23:18:20.630+11:002015-01-21T23:18:20.630+11:00Nick,
The filter you mention is an example of a F...Nick,<br /><br />The filter you mention is an example of a FIR filter, as such it 'should' have a finite response function.<br /><br />http://en.wikipedia.org/wiki/Savitzky%E2%80%93Golay_filter#Treatment_of_first_and_last_points<br /><br />The above suggested approach 'mirror image' is not a very good idea (at least for an end condition that is 'steep'), I usually try a double flip 'mirror image' but even there it becomes rather tricky (removing and adding a low order polynomial helps there)..<br /><br />I've used FFT (removing a lower polynomial first, and forcing the polynomial through both end points), but this has issues at the lowest and transition band frequencies.<br /><br />I've also used an IIR filter (two-pass Butterworth with zero padding, again removing a lower polynomial first, then adding it back afterwards, running at quad precision up to N = 80 pole count), this becomes really difficult for short time series as the filter literally rings forever (I'm 'still' working on that one).<br /><br />For the IIR filter, I use a 9-point FD stencil for the 1st and 2nd derivatives.<br /><br />Similarly, a FIR LOESS (n = 2 quadratic) 'can' be used (just don't do interpolation as the response is 'lumpy'), again I use a 9-point FD stencil for the 1st and 2nd derivatives.<br /><br />AFAIK, all filters suffer from the end point problem inherent in finite time series (you only have half the information at the end as you do in the middle).<br /><br />Don't have the necessary math skills, I always test with a very long white noise time series, chop the series up, and look at the various filter end effects.Everett F Sargenthttps://www.blogger.com/profile/00201577558036010680noreply@blogger.com