Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.graphics.apps.gnuplot > #1904 > unrolled thread

smooth kdensity

Back to article view | Back to comp.graphics.apps.gnuplot

Contents

  smooth kdensity Karl <mail.kfr@gmx.net> - 2013-06-20 15:05 +0200
    Re: smooth kdensity sfeam <sfeam@users.sourceforge.net> - 2013-06-20 08:28 -0700
      Re: smooth kdensity Karl <mail.kfr@gmx.net> - 2013-06-20 20:09 +0200
        Re: smooth kdensity sfeam <sfeam@users.sourceforge.net> - 2013-06-21 17:48 -0700
          Re: smooth kdensity Karl <mail.kfr@gmx.net> - 2013-06-22 16:34 +0200
            Re: smooth kdensity sfeam <sfeam@users.sourceforge.net> - 2013-06-22 12:48 -0700
              Re: smooth kdensity Karl <mail.kfr@gmx.net> - 2013-06-24 06:12 +0200
                Re: smooth kdensity sfeam <sfeam@users.sourceforge.net> - 2013-06-28 12:09 -0700

#1904 — smooth kdensity

Hi,

the help states that

 "If only two columns have been specified, or if the value of the third
column is zero or less, gnuplot calculates the bandwidth which would be
optimal if the input data was normally distributed. (This will usually
be a very conservative, i.e. broad bandwidth.) "

How exactly is the bandwidth (that is sigma?) chosen, and what makes
that "optimal"? I´ve tried to read up on "kernel desity estimation", and
found this here

http://en.wikipedia.org/wiki/Kernel_density_estimation#Practical_estimation_of_the_bandwidth

with "Silverman´s rule of thumb". Is that it?

And is there a way to read out the value afterwards for re-use, or
determine it otherwise?

 - Karl

[toc] | [next] | [standalone]

#1905

Karl wrote:

> Hi,
> 
> the help states that
> 
>  "If only two columns have been specified, or if the value of the
>  third
> column is zero or less, gnuplot calculates the bandwidth which would
> be optimal if the input data was normally distributed. (This will
> usually be a very conservative, i.e. broad bandwidth.) "
> 
> How exactly is the bandwidth (that is sigma?) chosen, and what makes
> that "optimal"? 

    /* This is the optimal bandwidth if the point distribution is Gaussian.
       (Applied Smoothing Techniques for Data Analysis
       by Adrian W, Bowman & Adelchi Azzalini (1997)) */
    default_bandwidth = pow( 4.0/(3.0*num_points), 1.0/5.0 )*sigma;


> I´ve tried to read up on "kernel desity estimation",
> and found this here
> 
> http://en.wikipedia.org/wiki/Kernel_density_estimation#Practical_estimation_of_the_bandwidth
> 
> with "Silverman´s rule of thumb". Is that it?
> 
> And is there a way to read out the value afterwards for re-use, or
> determine it otherwise?

  from "help kdensity":

  The bandwidth used in the previous plot is stored in variable 
  GPVAL_KDENSITY_BANDWIDTH.



> 
>  - Karl

[toc] | [prev] | [next] | [standalone]

#1906

Am 20.06.2013 17:28, schrieb sfeam:
> Karl wrote:
> 
>> How exactly is the bandwidth (that is sigma?) chosen, and what makes
>> that "optimal"? 
> 
>     /* This is the optimal bandwidth if the point distribution is Gaussian.
>        (Applied Smoothing Techniques for Data Analysis
>        by Adrian W, Bowman & Adelchi Azzalini (1997)) */
>     default_bandwidth = pow( 4.0/(3.0*num_points), 1.0/5.0 )*sigma;

OK, that is the formula given on wp. Should perhaps be explained in the
help:

... optimal for normally distributed input data. The value is
(4/3n)^(1/5) times the standard deviation of the input data (with n
being the number of points), following the so-called "Silverman´s rule
of thumb".

Plus perhaps the reference given above?

Btw., i just checked the CVS source, and it looks to me (im not good
with C) as if the the weight of the data points is not taken into
account for determining avg and sigma. Is that true, and if so, is it
intended?


And i saw that the possibility to give an explicit kernel width for
every data point in a third column has been removed in CVS. For example
for scattering experiments, the kernel should be larger for smaller
2theta values, because the resolution of a linear detector is smaller
there. I only found this newspost

http://newsgroups.derkeiler.com/Archive/Comp/comp.graphics.apps.gnuplot/2012-10/msg00007.html

about that. Could there be another solution to resolve this disambiguity
in what the third column contains?


Best,
  Karl

[toc] | [prev] | [next] | [standalone]

#1907

Karl wrote:

> And i saw that the possibility to give an explicit kernel width for
> every data point in a third column has been removed in CVS. For
> example for scattering experiments, the kernel should be larger for
> smaller 2theta values, because the resolution of a linear detector is
> smaller there. 

My understanding is that bandwidth is a property of all the points
jointly, so it would make no sense to have a different bandwidth
for each point.  Do you have a reference that discusses the use
of a non-constant bandwidth?

	Ethan

> http://newsgroups.derkeiler.com/Archive/Comp/comp.graphics.apps.gnuplot/2012-10/msg00007.html
> Could there be another solution to resolve this ambiguity in what
> the third column contains? 
> 
> Best,
>   Karl

[toc] | [prev] | [next] | [standalone]

#1908

Am 22.06.2013 02:48, schrieb sfeam:
> Karl wrote:
> 
>> And i saw that the possibility to give an explicit kernel width for
>> every data point in a third column has been removed in CVS. For
>> example for scattering experiments, the kernel should be larger for
>> smaller 2theta values, because the resolution of a linear detector is
>> smaller there. 
> 
> My understanding is that bandwidth is a property of all the points
> jointly, so it would make no sense to have a different bandwidth
> for each point.  Do you have a reference that discusses the use
> of a non-constant bandwidth?
> 
I´ll have to dig for some reference, tell you when i have one.

One thing is that in a diffraction pattern with several peaks, each can
have it´s idividual lineshape. But then i think they would have to be
handled (smoothed) individually, anyway.

But when the scattered points positions are not evenly distributed (as
is the case if you measure angles with a linear or planar detector
array), then the larger points should clearly get a proportionally
higher bandwidth, i´d say.

 Karl

[toc] | [prev] | [next] | [standalone]

#1910

Karl wrote:

> Am 22.06.2013 02:48, schrieb sfeam:
>> Karl wrote:
>> 
>>> And i saw that the possibility to give an explicit kernel width for
>>> every data point in a third column has been removed in CVS. For
>>> example for scattering experiments, the kernel should be larger for
>>> smaller 2theta values, because the resolution of a linear detector
>>> is smaller there.
>> 
>> My understanding is that bandwidth is a property of all the points
>> jointly, so it would make no sense to have a different bandwidth
>> for each point.  Do you have a reference that discusses the use
>> of a non-constant bandwidth?
>> 
> I´ll have to dig for some reference, tell you when i have one.
> 
> One thing is that in a diffraction pattern with several peaks, each
> can have it´s idividual lineshape. 

By "lineshape" do you mean the rocking (Darwin) curve?
Below you are talking about detector nonlinearity, but here I'm not
sure whether you are talking about a property of the source, the
sample, or what.

> But then i think they would have to
> be handled (smoothed) individually, anyway.
> 
> But when the scattered points positions are not evenly distributed (as
> is the case if you measure angles with a linear or planar detector
> array), then the larger points should clearly get a proportionally
> higher bandwidth, i´d say.

I'm not sure about that.  In that case isn't the proper approach to
describe the non-linearity of the axis explicitly, and do your smoothing
or other analysis on the corrected coordinates?  It's going to be
necessary to work back to angular coordinates in any case in order to
interpret the peak locations.

	Ethan

> 
>  Karl

[toc] | [prev] | [next] | [standalone]

#1911

Am 22.06.2013 21:48, schrieb sfeam:
> Karl wrote:
>> One thing is that in a diffraction pattern with several peaks, each
>> can have it´s idividual lineshape. 
> 
> By "lineshape" do you mean the rocking (Darwin) curve?
> Below you are talking about detector nonlinearity, but here I'm not
> sure whether you are talking about a property of the source, the
> sample, or what.
> 
I meant sample dependant. Each perodic structure that generates a peak
can have it´s own correlation lenght, occurrence of defects, etc.

>> But then i think they would have to
>> be handled (smoothed) individually, anyway.

>>
>> But when the scattered points positions are not evenly distributed (as
>> is the case if you measure angles with a linear or planar detector
>> array), then the larger points should clearly get a proportionally
>> higher bandwidth, i´d say.
> 
> I'm not sure about that.  In that case isn't the proper approach to
> describe the non-linearity of the axis explicitly, and do your smoothing
> or other analysis on the corrected coordinates?  It's going to be
> necessary to work back to angular coordinates in any case in order to
> interpret the peak locations.
> 
You mean to first linearize, then smooth, and then do any further
analyzing (calculate scattering angles, q-vectors, etc.)? Possibly. In
any case the "sum of gaussians" approach probably isn´t the correct way
to handle different detectors. This would call for to not only feed
coordinates, weights and bandwidths to the kdensity smoothing, but also
the kernel itself.
That´s probably asking to much for a plotting program. ;-)

I still say that it might be useful for some applications to give an
individual bandwidth to each point. Isn´t there another way to resolve
the disambiguity with the "with boxes" style? Or could it just be kept
as it was, with proper documentation?


Btw., have you checked if the routine indeed disregards the weights for
calculating the average and deviation? That a bug?

Best, Karl

[toc] | [prev] | [next] | [standalone]

#1923

Karl wrote:

> I still say that it might be useful for some applications to give an
> individual bandwidth to each point. Isn´t there another way to resolve
> the disambiguity with the "with boxes" style? Or could it just be kept
> as it was, with proper documentation?
  
> Btw., have you checked if the routine indeed disregards the weights for
> calculating the average and deviation? That a bug?

Weights are not used in calculating the mean or standard deviation.
A related issue is still open on the tracker at 
  https://sourceforge.net/p/gnuplot/bugs/1118/
Should the calculations use sigma or e.s.d.?  (sum/N or sum/(N+1))

The kdensity code was originally contributed by Philipp Janert. 
I would be happy to get patches or requests for specific changes,
but it would be nice if they were accompanied by pointers to 
relevant literature or standard texts justifying the request.

	Ethan

 
> Best, Karl

[toc] | [prev] | [standalone]

Back to top | Article view | comp.graphics.apps.gnuplot

csiph-web