Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]


Groups > comp.soft-sys.math.mathematica > #8271 > unrolled thread

Derivative of experimental data

Started byGabriel Landi <gtlandi@gmail.com>
First post2012-02-01 08:49 +0000
Last post2012-02-06 07:37 +0000
Articles 9 — 5 participants

Back to article view | Back to comp.soft-sys.math.mathematica


Contents

  Derivative of experimental data Gabriel Landi <gtlandi@gmail.com> - 2012-02-01 08:49 +0000
    Re: Derivative of experimental data Yves Klett <yves.klett@googlemail.com> - 2012-02-02 09:58 +0000
    Re: Derivative of experimental data Armand Tamzarian <mike.honeychurch@gmail.com> - 2012-02-02 10:02 +0000
      Re: Derivative of experimental data "Oleksandr Rasputinov" <oleksandr_rasputinov@ymail.com> - 2012-02-03 07:06 +0000
        Re: Derivative of experimental data Armand Tamzarian <mike.honeychurch@gmail.com> - 2012-02-04 11:35 +0000
    Re: Derivative of experimental data "Oleksandr Rasputinov" <oleksandr_rasputinov@ymail.com> - 2012-02-02 10:03 +0000
    Re: Derivative of experimental data danl@wolfram.com - 2012-02-03 07:12 +0000
      Re: Derivative of experimental data Armand Tamzarian <mike.honeychurch@gmail.com> - 2012-02-04 11:35 +0000
      Re: Derivative of experimental data Gabriel Landi <gtlandi@gmail.com> - 2012-02-06 07:37 +0000

#8271 — Derivative of experimental data

FromGabriel Landi <gtlandi@gmail.com>
Date2012-02-01 08:49 +0000
SubjectDerivative of experimental data
Message-ID<jgauad$eq8$1@smc.vnet.net>
Dear MathGroup users,

I have a question which is very important for my current research, and 
which involves not only Mathematica, but computer science in general.

I have experimental data which is not very noisy, a small example of 
which may be downloaded here. 

Basically I need to compute the derivative of this data. Here are my 
options so far:

Option 1: Finite differencing. Its terrible since the noise enhances 
dramatically.
Option 2: Fitting some arbitrary function. The problem is that the 
general functional form of the data changes from experiment to 
experiment, so it is not possible to find a function which fits 
adequately in all cases.
Option 3: Savitzky-Golay filters (self-implemented in Mathematica, based 
on the discussion in Numerical Recipes, 3rd Ed.). It doesn't seem to 
make much of a difference; probably because my data is not really that 
noisy.
Option 4: Smoothing Splines filter. I am currently using Mr. Ludsteck 
package HPFilter. So far it is by far the best outcome. However, it is 
not free of some wild oscillations that are clearly non-analytical and 
which are giving me quite the headache.

Any suggestions are more than welcome.
I really appreciate any help I can get.

Best regards,

Gabriel Landi

[toc] | [next] | [standalone]


#8294

FromYves Klett <yves.klett@googlemail.com>
Date2012-02-02 09:58 +0000
Message-ID<jgdmnr$qmb$1@smc.vnet.net>
In reply to#8271
Hi,

The HPFilter works pretty well for some of my smoothing cases, too.
Probalby not helpful, but did you try supplying your own smoothing
parameter to the HPFilter function? At some version switch I had
problems with the automatically chosen parameter so I started supplying
my own values. The algorithm responds rather benignly to roughly chosen
(trial and error) values.

Another thing would be to try your hand at a frequency-based filter
(e.g. using FourierDCT and such).

Two commercial applications also come to mind, the EDA package
(http://www.wolfram.com/products/applications/eda/)  and (possibly
completely O.T.T. for casual use) DataModeler by Evolved Analytics.

I'd like to see some more state-of-the-art smoothing techniques
implemented into the standard functionality of Mathematica.

Regards,
Yves

Am 01.02.2012 09:49, schrieb Gabriel Landi:
> Dear MathGroup users,
> 
> I have a question which is very important for my current research, and 
> which involves not only Mathematica, but computer science in general.
> 
> I have experimental data which is not very noisy, a small example of 
> which may be downloaded here. 
> 
> Basically I need to compute the derivative of this data. Here are my 
> options so far:
> 
> Option 1: Finite differencing. Its terrible since the noise enhances 
> dramatically.
> Option 2: Fitting some arbitrary function. The problem is that the 
> general functional form of the data changes from experiment to 
> experiment, so it is not possible to find a function which fits 
> adequately in all cases.
> Option 3: Savitzky-Golay filters (self-implemented in Mathematica, based 
> on the discussion in Numerical Recipes, 3rd Ed.). It doesn't seem to 
> make much of a difference; probably because my data is not really that 
> noisy.
> Option 4: Smoothing Splines filter. I am currently using Mr. Ludsteck 
> package HPFilter. So far it is by far the best outcome. However, it is 
> not free of some wild oscillations that are clearly non-analytical and 
> which are giving me quite the headache.
> 
> Any suggestions are more than welcome.
> I really appreciate any help I can get.
> 
> Best regards,
> 
> Gabriel Landi

[toc] | [prev] | [next] | [standalone]


#8297

FromArmand Tamzarian <mike.honeychurch@gmail.com>
Date2012-02-02 10:02 +0000
Message-ID<jgdmvj$qoj$1@smc.vnet.net>
In reply to#8271
On Feb 1, 7:49 pm, Gabriel Landi <gtla...@gmail.com> wrote:
> Dear MathGroup users,
>
> I have a question which is very important for my current research, and
> which involves not only Mathematica, but computer science in general.
>
> I have experimental data which is not very noisy, a small example of
> which may be downloaded here.
>
> Basically I need to compute the derivative of this data. Here are my
> options so far:
>
> Option 1: Finite differencing. Its terrible since the noise enhances
> dramatically.
> Option 2: Fitting some arbitrary function. The problem is that the
> general functional form of the data changes from experiment to
> experiment, so it is not possible to find a function which fits
> adequately in all cases.
> Option 3: Savitzky-Golay filters (self-implemented in Mathematica, based
> on the discussion in Numerical Recipes, 3rd Ed.). It doesn't seem to
> make much of a difference; probably because my data is not really that
> noisy.
> Option 4: Smoothing Splines filter. I am currently using Mr. Ludsteck
> package HPFilter. So far it is by far the best outcome. However, it is
> not free of some wild oscillations that are clearly non-analytical and
> which are giving me quite the headache.
>
> Any suggestions are more than welcome.
> I really appreciate any help I can get.
>
> Best regards,
>
> Gabriel Landi

taking the numerical derivative of experimental data is a noisy
process just as the reverse, integrating, is a smoothing process. that
is the way it is.

incidentally there is a Savitzky-Golay in the wolfram library but it
seems no one ever goes there now to search for code.

Mike

[toc] | [prev] | [next] | [standalone]


#8312

From"Oleksandr Rasputinov" <oleksandr_rasputinov@ymail.com>
Date2012-02-03 07:06 +0000
Message-ID<jgg12b$bl4$1@smc.vnet.net>
In reply to#8297
On Thu, 02 Feb 2012 10:02:27 -0000, Armand Tamzarian  
<mike.honeychurch@gmail.com> wrote:

>
> incidentally there is a Savitzky-Golay in the wolfram library but it
> seems no one ever goes there now to search for code.
>
> Mike
>

The code on MathSource is inefficiently written and completely ignores the  
endpoints, resulting in a truncated signal. The code I posted is better.  
Maybe I'll submit it to MathSource one day...

[toc] | [prev] | [next] | [standalone]


#8338

FromArmand Tamzarian <mike.honeychurch@gmail.com>
Date2012-02-04 11:35 +0000
Message-ID<jgj563$n2u$1@smc.vnet.net>
In reply to#8312
On Feb 3, 6:06 pm, "Oleksandr Rasputinov"
<oleksandr_rasputi...@ymail.com> wrote:
> On Thu, 02 Feb 2012 10:02:27 -0000, Armand Tamzarian
>
> <mike.honeychu...@gmail.com> wrote:
>
> > incidentally there is a Savitzky-Golay in the wolfram library but it
> > seems no one ever goes there now to search for code.
>
> > Mike
>
> The code on MathSource is inefficiently written and completely ignores th=
e
> endpoints, resulting in a truncated signal. The code I posted is better.
> Maybe I'll submit it to MathSource one day...

You should. MathSource seems to have been forgotten by both users and
Wolfram but should be the place for users to share code I think.

Mike

[toc] | [prev] | [next] | [standalone]


#8299

From"Oleksandr Rasputinov" <oleksandr_rasputinov@ymail.com>
Date2012-02-02 10:03 +0000
Message-ID<jgdn1g$qp8$1@smc.vnet.net>
In reply to#8271
On Wed, 01 Feb 2012 08:49:17 -0000, Gabriel Landi <gtlandi@gmail.com>  
wrote:

> Dear MathGroup users,
>
> I have a question which is very important for my current research, and
> which involves not only Mathematica, but computer science in general.
>
> I have experimental data which is not very noisy, a small example of
> which may be downloaded here.
>
> Basically I need to compute the derivative of this data. Here are my
> options so far:
>
> Option 1: Finite differencing. Its terrible since the noise enhances
> dramatically.
> Option 2: Fitting some arbitrary function. The problem is that the
> general functional form of the data changes from experiment to
> experiment, so it is not possible to find a function which fits
> adequately in all cases.
> Option 3: Savitzky-Golay filters (self-implemented in Mathematica, based
> on the discussion in Numerical Recipes, 3rd Ed.). It doesn't seem to
> make much of a difference; probably because my data is not really that
> noisy.
> Option 4: Smoothing Splines filter. I am currently using Mr. Ludsteck
> package HPFilter. So far it is by far the best outcome. However, it is
> not free of some wild oscillations that are clearly non-analytical and
> which are giving me quite the headache.
>
> Any suggestions are more than welcome.
> I really appreciate any help I can get.
>
> Best regards,
>
> Gabriel Landi

Ordinarily I would suggest the Savitzky-Golay method for this problem, but  
since you mention it is not really necessary for your data, perhaps you  
can use a straightforward interpolation?

len = 25;
data = Transpose[{Range[0., len]/len, RandomReal[{-1, 1}, len + 1]}];

ifun = Interpolation[
  data,
  InterpolationOrder -> 3,
  Method -> "Spline"
];

Show[{
  Plot[{ifun[x], 1/len ifun'[x]}, {x, 0, 1}],
  ListPlot[data, PlotStyle -> Red]
}] (* looks OK *)

If this doesn't work, I think the Savitzky-Golay filter is most probably  
the way to go, but you might need to adjust the filter window width for  
best results. In case it would be helpful, here is my implementation of  
the filter. Since I generally only need it for smoothing (i.e. without  
differentiation) I haven't spent any time figuring out the correct  
procedure for dealing with the end-points when taking derivatives,  
although it may work well enough as-is.

ClearAll[SavitzkyGolayHatMatrix];
SavitzkyGolayHatMatrix[
     windowHalfWidth_Integer, polynomialOrder_Integer,
     derivativeOrder_Integer,
     opts___
     ] /; 0 <= derivativeOrder <= polynomialOrder <=
     2 windowHalfWidth :=
   Block[{
     wprec =
      WorkingPrecision /. {opts}~Join~
        Options[SavitzkyGolayHatMatrix, WorkingPrecision],
     vandermondeMatrix
     },
    vandermondeMatrix = Table[
       If[i == j == 0, 1, i^j],
       {i, -windowHalfWidth, +windowHalfWidth},
       {j, 0, polynomialOrder}
       ]~N~wprec;
    (-1)^derivativeOrder derivativeOrder!
     Drop[vandermondeMatrix, None, -derivativeOrder].Drop[
       PseudoInverse[vandermondeMatrix], derivativeOrder]
    ];
Options[SavitzkyGolayHatMatrix] = {WorkingPrecision -> Infinity};
Attributes[SavitzkyGolayHatMatrix] = {NHoldAll};

ClearAll[SavitzkyGolayFilter];
SavitzkyGolayFilter[
     data_?VectorQ,
     windowHalfWidth_Integer, polynomialOrder_Integer,
     derivativeOrder_Integer,
     opts___
     ] /; 0 <= derivativeOrder <= polynomialOrder <=
     2 windowHalfWidth < Length[data] :=
   Block[{
     wprec =
      WorkingPrecision /. {opts}~Join~
        Options[SavitzkyGolayFilter, WorkingPrecision],
     dprec = Precision[data],
     ndat, hatMatrix
     },
    ndat = If[wprec < dprec, N[data, wprec], data];
    hatMatrix = SavitzkyGolayHatMatrix[
      windowHalfWidth, polynomialOrder, derivativeOrder,
      WorkingPrecision -> Min[wprec, dprec]
      ];
    Flatten[{
      ListConvolve[#, Take[ndat, 2 windowHalfWidth + 1]] & /@
       Reverse@Take[hatMatrix, -windowHalfWidth],
      ListConvolve[hatMatrix[[windowHalfWidth + 1]], ndat],
      ListConvolve[#, Take[ndat, -(2 windowHalfWidth + 1)]] & /@
       Reverse@Take[hatMatrix, windowHalfWidth]
      }]
    ];
Options[SavitzkyGolayFilter] = {WorkingPrecision -> Infinity};
Attributes[SavitzkyGolayFilter] = {NHoldRest};

Example:

data = Tan@TriangleWave@Range[0.20, 5.80, 0.01]; (* signal *)
data += RandomReal[{-0.05, 0.05}, Length[data]]; (* noise *)

ListPlot[
  SavitzkyGolayFilter[data,
   8 (* window half-width *),
   3 (* polynomial order *),
   1 (* derivative order *)
  ],
  Joined -> True, PlotRange -> All
] (* looks OK *)

For wide windows with high polynomial orders, calculating the filter  
coefficients by this method can be subject to numerical instability. If  
so, you can set WorkingPrecision to whatever value is necessary (up to and  
including Infinity, although this uses rational arithmetic and so is slow).

[toc] | [prev] | [next] | [standalone]


#8311

Fromdanl@wolfram.com
Date2012-02-03 07:12 +0000
Message-ID<jgg1c0$boq$1@smc.vnet.net>
In reply to#8271
Some URLs to past discussion are here:

http://forums.wolfram.com/mathgroup/archive/2012/Jan/msg00396.html

Upshot: Smooth interpolation (splines maybe) or Fourier methods might be good for this. Also I wonder if finite differencing, with more than just immediate neighbors, might be alright? (This assumes you only used the immediate neighbors; if you tried with more and it was no good then that's that.)

Daniel Lichtblau
Wolfram Research

[toc] | [prev] | [next] | [standalone]


#8337

FromArmand Tamzarian <mike.honeychurch@gmail.com>
Date2012-02-04 11:35 +0000
Message-ID<jgj554$n27$1@smc.vnet.net>
In reply to#8311
On Feb 3, 6:12 pm, d...@wolfram.com wrote:
> Some URLs to past discussion are here:
>
> http://forums.wolfram.com/mathgroup/archive/2012/Jan/msg00396.html
>
> Upshot: Smooth interpolation (splines maybe) or Fourier methods might be 
good for this. Also I wonder if finite differencing, with more than just immediate neighbors, might be alright? (This assumes you only used the immediate neighbors; if you tried with more and it was no good then that's that.)
>
> Daniel Lichtblau
> Wolfram Research

there are plenty of difference schemes outlined in the documentation:

tutorial/NDSolvePDE

but any differencing method (sans smoothing of some type) is going to
give noisy output (right??)

Mike

[toc] | [prev] | [next] | [standalone]


#8356

FromGabriel Landi <gtlandi@gmail.com>
Date2012-02-06 07:37 +0000
Message-ID<jgo00f$9v3$1@smc.vnet.net>
In reply to#8311
Dear All,

I really appreciate the discussion. Here are a few comments I have 
gathered so far:

The Savitzky-Golay filter of Mr. Oleksandr seems to give the most 
faithful (perhaps trustworthy) representation of the derivative, albeit 
not obviously yielding a smooth curve.
As suggested by Mr. Odisio, the GaussianFilter and DerivativeFilter 
implemented in Mathematica can give smooth curves, which may however be 
problematic at the end points (apparently there are Padding options, 
which I unfortunately have not yet tested).
Mr. Dahl, I was unable to use your package. I can produce the 
interpolating function with no problems, but when I ask to compute it's 
derivative, it freezes (the computation remains going indefinitely).

Again, thank you all for the tips.

Best regards,

Gabriel Landi

[toc] | [prev] | [standalone]


Back to top | Article view | comp.soft-sys.math.mathematica


csiph-web