Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]


Groups > comp.soft-sys.math.mathematica > #8706

Re: Kolmogorov Smirnov in two or more dimensions is in Mathematica 8.0.4

From Ray Koopman <koopman@sfu.ca>
Newsgroups comp.soft-sys.math.mathematica
Subject Re: Kolmogorov Smirnov in two or more dimensions is in Mathematica 8.0.4
Date 2012-02-23 10:48 +0000
Organization Steven M. Christensen and Associates, Inc and MathTensor, Inc.
Message-ID <ji55hg$77q$1@smc.vnet.net> (permalink)
References <ji2g4f$k8d$1@smc.vnet.net>

Show all headers | View raw


On Feb 22, 2:30 am, maria giovanna dainotti
<mariagiovannadaino...@yahoo.it> wrote:
> Dear Andy Ross,
> thanks for your suggestion. I applied the method
> DistributionFitTest[data1, data2,
> {"TestDataTable","SzekelyEnergy"},Method->{"MonteCarlo","MonteCarloSamples"->1000}]
> and it works, but when I try to minimize the function I have a problem.
>
> For[j = 1, j < n + 1,
>  fn = Function[{s, m},
>    x = Table[{Data1[[j, 1]] + s, Data1[[j, 2]] + m}, {j, 1, size}];
>    -2*Log[
>      DistributionFitTest[x, Data0, {"TestDataTable", "SzekelyEnergy"},
>         Method -> {"MonteCarlo", "MonteCarloSamples" -> 1000}][[1, 1,
>       2, 3]]]
>    ]]
>
> With Data1
> {{0.894606, 14.0865}, {0.811846, 13.7631}, {1.00287,
>   14.2577}, {0.798426, 13.8605}, {0.765556, 14.0297}, {1.06602,
>   14.144}, {0.965456, 14.2174}, {0.879326, 14.6311}, {0.831876,
>   14.006}, {0.695166, 14.2763}, {1.05657, 14.2558}, {1.05519,
>   13.7674}, {0.811626, 14.1137}, {0.910116, 13.9954}, {0.933376,
>   13.9232}, {0.870476, 14.2965}, {0.713326, 14.7321}, {1.13929,
>   13.9139}, {1.1878, 14.0376}, {0.998686, 13.9364}, {1.10588,
>   14.0096}, {1.06213, 14.2319}, {0.797846, 14.9199}, {0.768916,
>   14.5909}, {0.944476, 14.8957}, {1.09493, 14.6853}, {1.03171,
>   14.5485}, {0.838416, 15.1119}, {1.03875, 14.3372}, {0.695036,
>   15.0422}, {0.860336, 14.9023}, {0.903736, 15.1527}, {0.798196,
>   14.7923}, {0.695146, 15.5481}, {1.08506, 15.2012}, {1.07127,
>   15.2615}, {0.919576, 15.5636}, {1.01526, 15.4415}, {1.15206,
>   15.0784}, {0.844826, 15.6197}, {0.897866, 15.887}, {0.762816,
>   15.7318}, {0.867886, 15.743}, {0.991576, 15.7403}, {0.898346,
>   15.6946}, {0.996826, 15.6467}, {1.0194, 15.6681}, {0.830686,
>   15.5817}, {0.982806, 15.6863}, {0.883306, 16.3492}, {0.928786,
>   15.9975}, {0.932786, 15.9091}, {1.08748, 15.8808}, {0.963906,
>   16.4309}, {0.896236, 16.7413}, {0.750366, 16.7619}, {0.945936,
>   16.6388}, {1.064, 16.7059}, {0.947046, 17.0593}, {1.0258,
>   16.3197}, {0.973976, 16.9003}, {0.829206, 17.183}, {0.917956,
>   17.0182}, {0.954016, 16.8625}, {0.842906, 17.2803}, {1.10261,
>   17.0396}, {1.02155, 17.3819}, {0.671206, 17.5726}, {1.04596,
>   17.076}, {1.08985, 17.3047}, {1.05809, 17.2071}, {0.916836,
>   17.6237}, {0.815676, 17.5559}, {0.943866, 18.1943}, {1.02348,
>   16.8333}, {1.00346, 16.9493}, {1.15636, 17.0909}, {1.11935,
>   17.5003}}
>
> and Data0
>
> {{0.97304, 14.1829}, {0.98663, 14.1295}, {0.98284, 14.3172}, {0.81423,
>    14.3466}, {0.97303, 14.5966}, {0.87122, 14.8435}, {0.90252,
>   14.9036}, {0.81887, 15.1177}, {1.07722, 14.6849}, {0.86684,
>   15.6456}, {0.86664, 14.7034}, {0.78728, 15.0898}, {1.10336,
>   14.4085}, {1.12014, 14.7281}, {0.95923, 14.4988}, {0.89942,
>   15.3173}, {0.83841, 14.8422}, {0.99105, 14.8813}, {1.111,
>   14.5964}, {0.93255, 15.5019}, {1.03142, 14.8009}, {1.00661,
>   15.0827}, {0.93255, 15.3064}, {1.10023, 14.7189}, {0.8797,
>   15.1038}, {1.0013, 14.6755}, {0.87673, 15.0952}, {0.84131,
>   15.7345}, {1.06392, 15.3528}, {1.00138, 14.9835}, {0.77803,
>   15.4637}, {0.76795, 15.3611}, {0.98328, 15.1047}, {0.89193,
>   14.8321}, {0.72882, 15.909}, {0.77123, 15.7902}, {0.86218,
>   15.6637}, {0.84381, 15.536}, {0.99263, 15.8903}, {1.0805,
>   15.1453}, {0.85316, 15.7793}, {0.85186, 15.9119}, {1.10898,
>   15.7583}, {1.03365, 15.7393}, {0.84783, 16.1911}, {1.10979,
>   16.0031}, {1.05238, 16.015}, {0.90259, 16.4864}, {0.84963,
>   15.9818}, {1.09221, 16.0088}, {1.0443, 15.8326}, {0.8945,
>   16.1927}, {0.83015, 16.2776}, {0.7551, 16.5538}, {1.05947,
>   15.9138}, {1.06189, 15.6061}, {1.05889, 16.0743}, {0.85216,
>   16.1568}, {0.72597, 16.7657}, {0.93638, 16.3583}, {0.81968,
>   16.2711}, {0.95022, 16.3549}, {1.04536, 16.18}, {1.0786,
>   16.0967}, {0.9385, 15.8504}, {0.95024, 15.9338}, {0.76753,
>   17.1461}, {1.18224, 16.0437}, {0.96447, 16.4908}, {0.98235,
>   16.0892}, {1.06151, 16.699}, {0.79052, 16.5207}, {1.15863,
>   16.3223}, {1.00795, 16.2444}, {1.07284, 16.6536}, {1.04796,
>   16.865}, {0.84226, 16.7247}, {1.04712, 16.2673}}
>
> [...]

Here is an implementation of the Szekely-Energy comparison of your
two data sets. It saves some time by taking advantage of the fact
that your two sample sizes are equal. Perhaps more importantly, it
uses a centered orthonormal basis for the space of the pooled data.
This gives both variables equal opportunity to influence the test
statistic. If this is not done, the test statistic tends to ignore
variables with smaller variances.

Based on 10^6 samples, the approximate p-value is .0587 .

n1 = Length[data1 = {{0.894606, 14.0865}, ..., {1.11935, 17.5003}}]
n0 = Length[data0 = {{0.97304, 14.1829}, ..., {1.04712, 16.2673}}]
n = Length[data = Join[data1, data0]]

78
78
156

zz = Transpose@Orthogonalize[Transpose@data - Mean@data];
d = Outer[Norm[#1-#2]&, zz, zz, 1];
a = Total[Take[d, n1, -n0], 2]

883.529

reps = 1*^6
Timing[
  N@Total @ UnitStep[ Table[
  s = RandomSample[r = Range@n, n1]; r[[s]] = 0;
  t = SparseArray[r] /. SparseArray[_,_,_,x_] :> x[[3]];
  Total[d[[s,t]], 2], {reps}] - a] / reps ]

1000000

{3901.83, 0.058742}

Back to comp.soft-sys.math.mathematica | Previous | Next — Previous in thread | Find similar | Unroll thread


Thread

Re: Kolmogorov Smirnov in two or more dimensions is in Mathematica 8.0.4 maria giovanna dainotti <mariagiovannadainotti@yahoo.it> - 2012-02-22 10:30 +0000
  Re: Kolmogorov Smirnov in two or more dimensions is in Mathematica 8.0.4 Ray Koopman <koopman@sfu.ca> - 2012-02-23 10:48 +0000

csiph-web