Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > comp.soft-sys.math.mathematica > #8271 > unrolled thread
| Started by | Gabriel Landi <gtlandi@gmail.com> |
|---|---|
| First post | 2012-02-01 08:49 +0000 |
| Last post | 2012-02-06 07:37 +0000 |
| Articles | 9 — 5 participants |
Back to article view | Back to comp.soft-sys.math.mathematica
Derivative of experimental data Gabriel Landi <gtlandi@gmail.com> - 2012-02-01 08:49 +0000
Re: Derivative of experimental data Yves Klett <yves.klett@googlemail.com> - 2012-02-02 09:58 +0000
Re: Derivative of experimental data Armand Tamzarian <mike.honeychurch@gmail.com> - 2012-02-02 10:02 +0000
Re: Derivative of experimental data "Oleksandr Rasputinov" <oleksandr_rasputinov@ymail.com> - 2012-02-03 07:06 +0000
Re: Derivative of experimental data Armand Tamzarian <mike.honeychurch@gmail.com> - 2012-02-04 11:35 +0000
Re: Derivative of experimental data "Oleksandr Rasputinov" <oleksandr_rasputinov@ymail.com> - 2012-02-02 10:03 +0000
Re: Derivative of experimental data danl@wolfram.com - 2012-02-03 07:12 +0000
Re: Derivative of experimental data Armand Tamzarian <mike.honeychurch@gmail.com> - 2012-02-04 11:35 +0000
Re: Derivative of experimental data Gabriel Landi <gtlandi@gmail.com> - 2012-02-06 07:37 +0000
| From | Gabriel Landi <gtlandi@gmail.com> |
|---|---|
| Date | 2012-02-01 08:49 +0000 |
| Subject | Derivative of experimental data |
| Message-ID | <jgauad$eq8$1@smc.vnet.net> |
Dear MathGroup users, I have a question which is very important for my current research, and which involves not only Mathematica, but computer science in general. I have experimental data which is not very noisy, a small example of which may be downloaded here. Basically I need to compute the derivative of this data. Here are my options so far: Option 1: Finite differencing. Its terrible since the noise enhances dramatically. Option 2: Fitting some arbitrary function. The problem is that the general functional form of the data changes from experiment to experiment, so it is not possible to find a function which fits adequately in all cases. Option 3: Savitzky-Golay filters (self-implemented in Mathematica, based on the discussion in Numerical Recipes, 3rd Ed.). It doesn't seem to make much of a difference; probably because my data is not really that noisy. Option 4: Smoothing Splines filter. I am currently using Mr. Ludsteck package HPFilter. So far it is by far the best outcome. However, it is not free of some wild oscillations that are clearly non-analytical and which are giving me quite the headache. Any suggestions are more than welcome. I really appreciate any help I can get. Best regards, Gabriel Landi
[toc] | [next] | [standalone]
| From | Yves Klett <yves.klett@googlemail.com> |
|---|---|
| Date | 2012-02-02 09:58 +0000 |
| Message-ID | <jgdmnr$qmb$1@smc.vnet.net> |
| In reply to | #8271 |
Hi, The HPFilter works pretty well for some of my smoothing cases, too. Probalby not helpful, but did you try supplying your own smoothing parameter to the HPFilter function? At some version switch I had problems with the automatically chosen parameter so I started supplying my own values. The algorithm responds rather benignly to roughly chosen (trial and error) values. Another thing would be to try your hand at a frequency-based filter (e.g. using FourierDCT and such). Two commercial applications also come to mind, the EDA package (http://www.wolfram.com/products/applications/eda/) and (possibly completely O.T.T. for casual use) DataModeler by Evolved Analytics. I'd like to see some more state-of-the-art smoothing techniques implemented into the standard functionality of Mathematica. Regards, Yves Am 01.02.2012 09:49, schrieb Gabriel Landi: > Dear MathGroup users, > > I have a question which is very important for my current research, and > which involves not only Mathematica, but computer science in general. > > I have experimental data which is not very noisy, a small example of > which may be downloaded here. > > Basically I need to compute the derivative of this data. Here are my > options so far: > > Option 1: Finite differencing. Its terrible since the noise enhances > dramatically. > Option 2: Fitting some arbitrary function. The problem is that the > general functional form of the data changes from experiment to > experiment, so it is not possible to find a function which fits > adequately in all cases. > Option 3: Savitzky-Golay filters (self-implemented in Mathematica, based > on the discussion in Numerical Recipes, 3rd Ed.). It doesn't seem to > make much of a difference; probably because my data is not really that > noisy. > Option 4: Smoothing Splines filter. I am currently using Mr. Ludsteck > package HPFilter. So far it is by far the best outcome. However, it is > not free of some wild oscillations that are clearly non-analytical and > which are giving me quite the headache. > > Any suggestions are more than welcome. > I really appreciate any help I can get. > > Best regards, > > Gabriel Landi
[toc] | [prev] | [next] | [standalone]
| From | Armand Tamzarian <mike.honeychurch@gmail.com> |
|---|---|
| Date | 2012-02-02 10:02 +0000 |
| Message-ID | <jgdmvj$qoj$1@smc.vnet.net> |
| In reply to | #8271 |
On Feb 1, 7:49 pm, Gabriel Landi <gtla...@gmail.com> wrote: > Dear MathGroup users, > > I have a question which is very important for my current research, and > which involves not only Mathematica, but computer science in general. > > I have experimental data which is not very noisy, a small example of > which may be downloaded here. > > Basically I need to compute the derivative of this data. Here are my > options so far: > > Option 1: Finite differencing. Its terrible since the noise enhances > dramatically. > Option 2: Fitting some arbitrary function. The problem is that the > general functional form of the data changes from experiment to > experiment, so it is not possible to find a function which fits > adequately in all cases. > Option 3: Savitzky-Golay filters (self-implemented in Mathematica, based > on the discussion in Numerical Recipes, 3rd Ed.). It doesn't seem to > make much of a difference; probably because my data is not really that > noisy. > Option 4: Smoothing Splines filter. I am currently using Mr. Ludsteck > package HPFilter. So far it is by far the best outcome. However, it is > not free of some wild oscillations that are clearly non-analytical and > which are giving me quite the headache. > > Any suggestions are more than welcome. > I really appreciate any help I can get. > > Best regards, > > Gabriel Landi taking the numerical derivative of experimental data is a noisy process just as the reverse, integrating, is a smoothing process. that is the way it is. incidentally there is a Savitzky-Golay in the wolfram library but it seems no one ever goes there now to search for code. Mike
[toc] | [prev] | [next] | [standalone]
| From | "Oleksandr Rasputinov" <oleksandr_rasputinov@ymail.com> |
|---|---|
| Date | 2012-02-03 07:06 +0000 |
| Message-ID | <jgg12b$bl4$1@smc.vnet.net> |
| In reply to | #8297 |
On Thu, 02 Feb 2012 10:02:27 -0000, Armand Tamzarian <mike.honeychurch@gmail.com> wrote: > > incidentally there is a Savitzky-Golay in the wolfram library but it > seems no one ever goes there now to search for code. > > Mike > The code on MathSource is inefficiently written and completely ignores the endpoints, resulting in a truncated signal. The code I posted is better. Maybe I'll submit it to MathSource one day...
[toc] | [prev] | [next] | [standalone]
| From | Armand Tamzarian <mike.honeychurch@gmail.com> |
|---|---|
| Date | 2012-02-04 11:35 +0000 |
| Message-ID | <jgj563$n2u$1@smc.vnet.net> |
| In reply to | #8312 |
On Feb 3, 6:06 pm, "Oleksandr Rasputinov" <oleksandr_rasputi...@ymail.com> wrote: > On Thu, 02 Feb 2012 10:02:27 -0000, Armand Tamzarian > > <mike.honeychu...@gmail.com> wrote: > > > incidentally there is a Savitzky-Golay in the wolfram library but it > > seems no one ever goes there now to search for code. > > > Mike > > The code on MathSource is inefficiently written and completely ignores th= e > endpoints, resulting in a truncated signal. The code I posted is better. > Maybe I'll submit it to MathSource one day... You should. MathSource seems to have been forgotten by both users and Wolfram but should be the place for users to share code I think. Mike
[toc] | [prev] | [next] | [standalone]
| From | "Oleksandr Rasputinov" <oleksandr_rasputinov@ymail.com> |
|---|---|
| Date | 2012-02-02 10:03 +0000 |
| Message-ID | <jgdn1g$qp8$1@smc.vnet.net> |
| In reply to | #8271 |
On Wed, 01 Feb 2012 08:49:17 -0000, Gabriel Landi <gtlandi@gmail.com>
wrote:
> Dear MathGroup users,
>
> I have a question which is very important for my current research, and
> which involves not only Mathematica, but computer science in general.
>
> I have experimental data which is not very noisy, a small example of
> which may be downloaded here.
>
> Basically I need to compute the derivative of this data. Here are my
> options so far:
>
> Option 1: Finite differencing. Its terrible since the noise enhances
> dramatically.
> Option 2: Fitting some arbitrary function. The problem is that the
> general functional form of the data changes from experiment to
> experiment, so it is not possible to find a function which fits
> adequately in all cases.
> Option 3: Savitzky-Golay filters (self-implemented in Mathematica, based
> on the discussion in Numerical Recipes, 3rd Ed.). It doesn't seem to
> make much of a difference; probably because my data is not really that
> noisy.
> Option 4: Smoothing Splines filter. I am currently using Mr. Ludsteck
> package HPFilter. So far it is by far the best outcome. However, it is
> not free of some wild oscillations that are clearly non-analytical and
> which are giving me quite the headache.
>
> Any suggestions are more than welcome.
> I really appreciate any help I can get.
>
> Best regards,
>
> Gabriel Landi
Ordinarily I would suggest the Savitzky-Golay method for this problem, but
since you mention it is not really necessary for your data, perhaps you
can use a straightforward interpolation?
len = 25;
data = Transpose[{Range[0., len]/len, RandomReal[{-1, 1}, len + 1]}];
ifun = Interpolation[
data,
InterpolationOrder -> 3,
Method -> "Spline"
];
Show[{
Plot[{ifun[x], 1/len ifun'[x]}, {x, 0, 1}],
ListPlot[data, PlotStyle -> Red]
}] (* looks OK *)
If this doesn't work, I think the Savitzky-Golay filter is most probably
the way to go, but you might need to adjust the filter window width for
best results. In case it would be helpful, here is my implementation of
the filter. Since I generally only need it for smoothing (i.e. without
differentiation) I haven't spent any time figuring out the correct
procedure for dealing with the end-points when taking derivatives,
although it may work well enough as-is.
ClearAll[SavitzkyGolayHatMatrix];
SavitzkyGolayHatMatrix[
windowHalfWidth_Integer, polynomialOrder_Integer,
derivativeOrder_Integer,
opts___
] /; 0 <= derivativeOrder <= polynomialOrder <=
2 windowHalfWidth :=
Block[{
wprec =
WorkingPrecision /. {opts}~Join~
Options[SavitzkyGolayHatMatrix, WorkingPrecision],
vandermondeMatrix
},
vandermondeMatrix = Table[
If[i == j == 0, 1, i^j],
{i, -windowHalfWidth, +windowHalfWidth},
{j, 0, polynomialOrder}
]~N~wprec;
(-1)^derivativeOrder derivativeOrder!
Drop[vandermondeMatrix, None, -derivativeOrder].Drop[
PseudoInverse[vandermondeMatrix], derivativeOrder]
];
Options[SavitzkyGolayHatMatrix] = {WorkingPrecision -> Infinity};
Attributes[SavitzkyGolayHatMatrix] = {NHoldAll};
ClearAll[SavitzkyGolayFilter];
SavitzkyGolayFilter[
data_?VectorQ,
windowHalfWidth_Integer, polynomialOrder_Integer,
derivativeOrder_Integer,
opts___
] /; 0 <= derivativeOrder <= polynomialOrder <=
2 windowHalfWidth < Length[data] :=
Block[{
wprec =
WorkingPrecision /. {opts}~Join~
Options[SavitzkyGolayFilter, WorkingPrecision],
dprec = Precision[data],
ndat, hatMatrix
},
ndat = If[wprec < dprec, N[data, wprec], data];
hatMatrix = SavitzkyGolayHatMatrix[
windowHalfWidth, polynomialOrder, derivativeOrder,
WorkingPrecision -> Min[wprec, dprec]
];
Flatten[{
ListConvolve[#, Take[ndat, 2 windowHalfWidth + 1]] & /@
Reverse@Take[hatMatrix, -windowHalfWidth],
ListConvolve[hatMatrix[[windowHalfWidth + 1]], ndat],
ListConvolve[#, Take[ndat, -(2 windowHalfWidth + 1)]] & /@
Reverse@Take[hatMatrix, windowHalfWidth]
}]
];
Options[SavitzkyGolayFilter] = {WorkingPrecision -> Infinity};
Attributes[SavitzkyGolayFilter] = {NHoldRest};
Example:
data = Tan@TriangleWave@Range[0.20, 5.80, 0.01]; (* signal *)
data += RandomReal[{-0.05, 0.05}, Length[data]]; (* noise *)
ListPlot[
SavitzkyGolayFilter[data,
8 (* window half-width *),
3 (* polynomial order *),
1 (* derivative order *)
],
Joined -> True, PlotRange -> All
] (* looks OK *)
For wide windows with high polynomial orders, calculating the filter
coefficients by this method can be subject to numerical instability. If
so, you can set WorkingPrecision to whatever value is necessary (up to and
including Infinity, although this uses rational arithmetic and so is slow).
[toc] | [prev] | [next] | [standalone]
| From | danl@wolfram.com |
|---|---|
| Date | 2012-02-03 07:12 +0000 |
| Message-ID | <jgg1c0$boq$1@smc.vnet.net> |
| In reply to | #8271 |
Some URLs to past discussion are here: http://forums.wolfram.com/mathgroup/archive/2012/Jan/msg00396.html Upshot: Smooth interpolation (splines maybe) or Fourier methods might be good for this. Also I wonder if finite differencing, with more than just immediate neighbors, might be alright? (This assumes you only used the immediate neighbors; if you tried with more and it was no good then that's that.) Daniel Lichtblau Wolfram Research
[toc] | [prev] | [next] | [standalone]
| From | Armand Tamzarian <mike.honeychurch@gmail.com> |
|---|---|
| Date | 2012-02-04 11:35 +0000 |
| Message-ID | <jgj554$n27$1@smc.vnet.net> |
| In reply to | #8311 |
On Feb 3, 6:12 pm, d...@wolfram.com wrote: > Some URLs to past discussion are here: > > http://forums.wolfram.com/mathgroup/archive/2012/Jan/msg00396.html > > Upshot: Smooth interpolation (splines maybe) or Fourier methods might be good for this. Also I wonder if finite differencing, with more than just immediate neighbors, might be alright? (This assumes you only used the immediate neighbors; if you tried with more and it was no good then that's that.) > > Daniel Lichtblau > Wolfram Research there are plenty of difference schemes outlined in the documentation: tutorial/NDSolvePDE but any differencing method (sans smoothing of some type) is going to give noisy output (right??) Mike
[toc] | [prev] | [next] | [standalone]
| From | Gabriel Landi <gtlandi@gmail.com> |
|---|---|
| Date | 2012-02-06 07:37 +0000 |
| Message-ID | <jgo00f$9v3$1@smc.vnet.net> |
| In reply to | #8311 |
Dear All, I really appreciate the discussion. Here are a few comments I have gathered so far: The Savitzky-Golay filter of Mr. Oleksandr seems to give the most faithful (perhaps trustworthy) representation of the derivative, albeit not obviously yielding a smooth curve. As suggested by Mr. Odisio, the GaussianFilter and DerivativeFilter implemented in Mathematica can give smooth curves, which may however be problematic at the end points (apparently there are Padding options, which I unfortunately have not yet tested). Mr. Dahl, I was unable to use your package. I can produce the interpolating function with no problems, but when I ask to compute it's derivative, it freezes (the computation remains going indefinitely). Again, thank you all for the tips. Best regards, Gabriel Landi
[toc] | [prev] | [standalone]
Back to top | Article view | comp.soft-sys.math.mathematica
csiph-web