Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]


Groups > comp.soft-sys.math.mathematica > #11237 > unrolled thread

Trying to quickly split lists at the point of maximum variance reduction

Started byEarl Mitchell <earl.j.mitchell@gmail.com>
First post2012-08-06 00:39 +0000
Last post2012-08-10 06:48 +0000
Articles 3 — 2 participants

Back to article view | Back to comp.soft-sys.math.mathematica


Contents

  Trying to quickly split lists at the point of maximum variance reduction Earl Mitchell <earl.j.mitchell@gmail.com> - 2012-08-06 00:39 +0000
    Re: Trying to quickly split lists at the point of maximum variance reduction Ray Koopman <koopman@sfu.ca> - 2012-08-09 01:35 +0000
    Re: Trying to quickly split lists at the point of Earl Mitchell <earl.j.mitchell@gmail.com> - 2012-08-10 06:48 +0000

#11237 — Trying to quickly split lists at the point of maximum variance reduction

FromEarl Mitchell <earl.j.mitchell@gmail.com>
Date2012-08-06 00:39 +0000
SubjectTrying to quickly split lists at the point of maximum variance reduction
Message-ID<jvn3om$am6$1@smc.vnet.net>
Hi all,

I've got a list of 42,000 lists of 233 variables (integers from 1 to 250),
each with a response in the last position (integer from 0 to 9).  I want to
find the variable and split point in that variable at which the weighted
average variance in the response will be minimized.

As a trivial example say the original variance in the response for the
whole set is 100.  Imagine that there is some variable, say at position
220, which has Tally'd values of {{0,40,000},{1,2000}} and, if we split the
corresponding responses along this break point the weighted average
variance of the two lists is 80, which happens to be the lowest possible
resulting variance for any single split, on any single variable.

How can I find this point quickly?  This description might be confusing -
let me know if something is not clear.  Thanks ahead of time for the help!

Mitch

PS.  Currently this job can be done in with this code:

FindMaxVarianceReductionSplit[data_List] :=
 Module[{transdata, splitvarreductionpairs, withoutputs, testsplits, split,
   endvar, maxvarreduction},
  transdata = Transpose[data];
  splitvarreductionpairs = With[{outputs = transdata[[-1]]},
    ParallelTable[
     With[{inputs = transdata[[i]], startvar = N@NewVariance[outputs]},
      withoutputs = Thread[{inputs, outputs}];
      testsplits = Union[inputs];
      Table[
       With[{splitval = testsplits[[j]]},
        split = GatherBy[withoutputs, #[[1]] > splitval &];
        endvar =
         Total[(N@Length[#]*NewVariance[#[[All, -1]]] & /@ split)/
           Length[withoutputs]];
        {splitval, startvar - endvar}
        ]
       ,
       {j, Length[testsplits]}]], {i, Length[Most[transdata]]}]
    ];

  maxvarreduction = Max[Flatten[splitvarreductionpairs, 1][[All, -1]]];
  Position[splitvarreductionpairs, maxvarreduction]

  ]


... on my brand spanking new MBP it completes in just under 1,000 seconds
being parallelized.  I need this to run much faster to have any practical
applications.


Thanks again!

[toc] | [next] | [standalone]


#11280

FromRay Koopman <koopman@sfu.ca>
Date2012-08-09 01:35 +0000
Message-ID<jvv447$ppt$1@smc.vnet.net>
In reply to#11237
On Aug 5, 5:39 pm, Earl Mitchell <earl.j.mitch...@gmail.com> wrote:
> Hi all,
>
> I've got a list of 42,000 lists of 233 variables (integers from 1 to 250),
> each with a response in the last position (integer from 0 to 9).  I want to
> find the variable and split point in that variable at which the weighted
> average variance in the response will be minimized.
>
> As a trivial example say the original variance in the response for the
> whole set is 100.  Imagine that there is some variable, say at position
> 220, which has Tally'd values of {{0,40,000},{1,2000}} and, if we split the
> corresponding responses along this break point the weighted average
> variance of the two lists is 80, which happens to be the lowest possible
> resulting variance for any single split, on any single variable.
>
> How can I find this point quickly?  This description might be confusing -
> let me know if something is not clear.  Thanks ahead of time for the help!
>
> Mitch
>
> PS.  Currently this job can be done in with this code:
>
> FindMaxVarianceReductionSplit[data_List] :=
>  Module[{transdata, splitvarreductionpairs, withoutputs, testsplits, split,
>    endvar, maxvarreduction},
>   transdata = Transpose[data];
>   splitvarreductionpairs = With[{outputs = transdata[[-1]]},
>     ParallelTable[
>      With[{inputs = transdata[[i]], startvar = N@NewVariance[outputs]},
>       withoutputs = Thread[{inputs, outputs}];
>       testsplits = Union[inputs];
>       Table[
>        With[{splitval = testsplits[[j]]},
>         split = GatherBy[withoutputs, #[[1]] > splitval &];
>         endvar =
>          Total[(N@Length[#]*NewVariance[#[[All, -1]]] & /@ split)/
>            Length[withoutputs]];
>         {splitval, startvar - endvar}
>         ]
>        ,
>        {j, Length[testsplits]}]], {i, Length[Most[transdata]]}]
>     ];
>
>   maxvarreduction = Max[Flatten[splitvarreductionpairs, 1][[All, -1]]];
>   Position[splitvarreductionpairs, maxvarreduction]
>
>   ]
>
> ... on my brand spanking new MBP it completes in just under 1,000 seconds
> being parallelized.  I need this to run much faster to have any practical
> applications.
>
> Thanks again!

I find your description confusing. Here's how I read it:

You have a table whose dimensions are 42000 x 233, and a corresponding
vector of 42000 responses. All the values in the table are integers in
[1, 250]. All the responses are integers in [0,9]. You want to choose
a column, say k, and a value, say x, such that if you split the
responses into two groups according as the k'th element in the
corresponding row of the table is <= x or > x, you minimize the sum of
squared deviations of the responses from their respective group
means.

Please confirm or correct that interpretation.

[toc] | [prev] | [next] | [standalone]


#11309 — Re: Trying to quickly split lists at the point of

FromEarl Mitchell <earl.j.mitchell@gmail.com>
Date2012-08-10 06:48 +0000
SubjectRe: Trying to quickly split lists at the point of
Message-ID<k02ara$2v7$1@smc.vnet.net>
In reply to#11237
That is spot on.  Sorry for the clumsy description.

For some context (since you may know other resources to suggest along these
lines) this is part of a larger project to create a suite of data-mining
and machine learning tools, with the goal being three fold: increase my
mathematica programming abilities, increase my knowledge of advanced
statistical methods, and (obviously) generate some algorithms that can be
used to make predictions.

This particular step is part of the RandomForrest statistical method.

Thanks so much for your help,
Mitch

On Wed, Aug 8, 2012 at 7:34 PM, Ray Koopman <koopman@sfu.ca> wrote:

> On Aug 5, 5:39 pm, Earl Mitchell <earl.j.mitch...@gmail.com> wrote:
> > Hi all,
> >
> > I've got a list of 42,000 lists of 233 variables (integers from 1 to
> 250),
> > each with a response in the last position (integer from 0 to 9).  I want
> to
> > find the variable and split point in that variable at which the weighted
> > average variance in the response will be minimized.
> >
> > As a trivial example say the original variance in the response for the
> > whole set is 100.  Imagine that there is some variable, say at position
> > 220, which has Tally'd values of {{0,40,000},{1,2000}} and, if we split
> the
> > corresponding responses along this break point the weighted average
> > variance of the two lists is 80, which happens to be the lowest possible
> > resulting variance for any single split, on any single variable.
> >
> > How can I find this point quickly?  This description might be confusing -
> > let me know if something is not clear.  Thanks ahead of time for the
> help!
> >
> > Mitch
> >
> > PS.  Currently this job can be done in with this code:
> >
> > FindMaxVarianceReductionSplit[data_List] :=
> >  Module[{transdata, splitvarreductionpairs, withoutputs, testsplits,
> split,
> >    endvar, maxvarreduction},
> >   transdata = Transpose[data];
> >   splitvarreductionpairs = With[{outputs = transdata[[-1]]},
> >     ParallelTable[
> >      With[{inputs = transdata[[i]], startvar = N@NewVariance[outputs]},
> >       withoutputs = Thread[{inputs, outputs}];
> >       testsplits = Union[inputs];
> >       Table[
> >        With[{splitval = testsplits[[j]]},
> >         split = GatherBy[withoutputs, #[[1]] > splitval &];
> >         endvar =
> >          Total[(N@Length[#]*NewVariance[#[[All, -1]]] & /@ split)/
> >            Length[withoutputs]];
> >         {splitval, startvar - endvar}
> >         ]
> >        ,
> >        {j, Length[testsplits]}]], {i, Length[Most[transdata]]}]
> >     ];
> >
> >   maxvarreduction = Max[Flatten[splitvarreductionpairs, 1][[All, -1]]];
> >   Position[splitvarreductionpairs, maxvarreduction]
> >
> >   ]
> >
> > ... on my brand spanking new MBP it completes in just under 1,000 seconds
> > being parallelized.  I need this to run much faster to have any practical
> > applications.
> >
> > Thanks again!
>
> I find your description confusing. Here's how I read it:
>
> You have a table whose dimensions are 42000 x 233, and a corresponding
> vector of 42000 responses. All the values in the table are integers in
> [1, 250]. All the responses are integers in [0,9]. You want to choose
> a column, say k, and a value, say x, such that if you split the
> responses into two groups according as the k'th element in the
> corresponding row of the table is <= x or > x, you minimize the sum of
> squared deviations of the responses from their respective group
> means.
>
> Please confirm or correct that interpretation.
>
>

[toc] | [prev] | [standalone]


Back to top | Article view | comp.soft-sys.math.mathematica


csiph-web