Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #109700 > unrolled thread

which library has map reduce and how to use it for this case

Back to article view | Back to comp.lang.python

Contents

  which library has map reduce and how to use it for this case Ho Yeung Lee <davidbenny2000@gmail.com> - 2016-06-08 23:52 -0700
    Re: which library has map reduce and how to use it for this case Michael Selik <michael.selik@gmail.com> - 2016-06-09 22:44 +0000
      Re: which library has map reduce and how to use it for this case Ho Yeung Lee <davidbenny2000@gmail.com> - 2016-06-09 18:07 -0700
        Re: which library has map reduce and how to use it for this case Michael Selik <michael.selik@gmail.com> - 2016-06-10 14:03 +0000
          Re: which library has map reduce and how to use it for this case meInvent bbird <jobmattcon@gmail.com> - 2016-06-10 23:19 -0700
            Re: which library has map reduce and how to use it for this case Michael Selik <michael.selik@gmail.com> - 2016-06-11 16:44 +0000

#109700 — which library has map reduce and how to use it for this case

i got M1 to M5 and V6 operator

and would like to do a full combination and result will also act among each other, 

map reduce attract my application

how to use this in this example?

actually below is like vlookup
then example is op3(op2(op1(x->y, y->z)), x->z)

search which combinations will result in a column result is 1


M1 = {}
M2 = {}
M3 = {}
M4 = {}
M5 = {}
V6 = {}
M1['00']=0
M1['01']=1
M1['02']=1
M1['10']=1
M1['11']=1
M1['12']=1
M1['20']=1
M1['21']=1
M1['22']=1
M2['00']=0
M2['01']=0
M2['02']=1
M2['10']=0
M2['11']=1
M2['12']=1
M2['20']=1
M2['21']=1
M2['22']=1
M3['00']=0
M3['01']=0
M3['02']=0
M3['10']=0
M3['11']=1
M3['12']=1
M3['20']=0
M3['21']=1
M3['22']=1
M4['00']=0
M4['01']=1
M4['02']=0
M4['10']=1
M4['11']=1
M4['12']=1
M4['20']=0
M4['21']=1
M4['22']=1
M5['00']=0
M5['01']=0
M5['02']=0
M5['10']=0
M5['11']=1
M5['12']=1
M5['20']=0
M5['21']=1
M5['22']=1
V6['00']=1
V6['01']=1
V6['02']=1
V6['10']=1
V6['11']=1
V6['12']=1
V6['20']=0
V6['21']=1
V6['22']=1
MM = {}
MM[0] = M1
MM[1] = M2
MM[2] = M3
MM[3] = M4
MM[4] = M5
MM[5] = V6
m = 3
b = [str(i)+str(j)+str(k) for i in range(m) for j in range(m) for k in range(m)]
b[21][0:1]+b[21][1:2]
b[21][1:2]+b[21][2:3]
b[21][0:1]+b[21][2:3]

#def node(mmm, A):
 #H2 = [MM[mmm][b[i][0:1]+b[i][1:2]] for i in range(len(b))]
 #return H2

#node(5, b[i][0:1]+b[i][1:2])


H2 = [MM[5][b[i][0:1]+b[i][1:2]] for i in range(len(b))]
H3 = [MM[5][b[i][0:1]+b[i][1:2]] for i in range(len(b))]
[str(k)+str(v) for k, v in zip(H2, H3)]

import itertools
deep = 3

finalresult = []
def DFS(H2, H3, b, deep, mresult)
 mylist = [i for i in range(0,7-1)]
 for aa,bb in itertools.combinations(mylist, 2):
  print(aa,bb)
  if deep == 3:
   #op0, op1
   op1xy = [MM[aa][b[i][0:1]+b[i][1:2]] for i in range(len(b))]
   op1yz = [MM[aa][b[i][1:2]+b[i][2:3]] for i in range(len(b))]
   op1xz = [MM[aa][b[i][0:1]+b[i][2:3]] for i in range(len(b))]
   op2xy = [MM[bb][b[i][0:1]+b[i][1:2]] for i in range(len(b))]
   op2yz = [MM[bb][b[i][1:2]+b[i][2:3]] for i in range(len(b))]
   op2xz = [MM[bb][b[i][0:1]+b[i][2:3]] for i in range(len(b))]
   mresult.append(op1xy)
   mresult.append(op1yz)
   mresult.append(op1xz)
   mresult.append(op2xy)
   mresult.append(op2yz)
   mresult.append(op2xz)
  else:
   H1 = H2
   H9 = H3
  ba = b
  b = [str(k)+str(v) for k, v in zip(H2, H3)]
  DFS(H1, H9, b)
  b = ba

DFS(0, 0, 0, 3, finalresult)

[toc] | [next] | [standalone]

#109766

I like using Yelp's mrjob module (https://github.com/Yelp/mrjob) to run
Python on Hadoop.

On Thu, Jun 9, 2016 at 2:56 AM Ho Yeung Lee <davidbenny2000@gmail.com>
wrote:

> [... a bunch of code ...]


If you want to describe a map-reduce problem, start with the data. What
does a record of your input data look like?

Then think about your mapper. What key-value pairs will you extract from
each line of data?

Then think about your reducer. For a single key and its associated values,
what will you calculate?

[toc] | [prev] | [next] | [standalone]

#109769

input are these six operators, output is finding full column of 27 elements add together is 54

i got memory error when searching this, 

i have difficulty in understanding map reduce and transforming my program into map reduce problem

M1 = {}
M2 = {}
M3 = {}
M4 = {}
M5 = {}
V6 = {}
M1['00']=0
M1['01']=2
M1['02']=1
M1['10']=1
M1['11']=1
M1['12']=1
M1['20']=1
M1['21']=1
M1['22']=2
M2['00']=0
M2['01']=1
M2['02']=1
M2['10']=1
M2['11']=1
M2['12']=1
M2['20']=1
M2['21']=1
M2['22']=1
M3['00']=2
M3['01']=2
M3['02']=2
M3['10']=0
M3['11']=2
M3['12']=1
M3['20']=0
M3['21']=1
M3['22']=2
M4['00']=1
M4['01']=2
M4['02']=1
M4['10']=2
M4['11']=2
M4['12']=2
M4['20']=0
M4['21']=1
M4['22']=2
M5['00']=0
M5['01']=1
M5['02']=1
M5['10']=0
M5['11']=2
M5['12']=1
M5['20']=0
M5['21']=1
M5['22']=1
V6['00']=1
V6['01']=1
V6['02']=2
V6['10']=1
V6['11']=2
V6['12']=1
V6['20']=1
V6['21']=2
V6['22']=2
MM = {}
MM[0] = M1
MM[1] = M2
MM[2] = M3
MM[3] = M4
MM[4] = M5
MM[5] = V6
m = 3
b = [str(i)+str(j)+str(k) for i in range(m) for j in range(m) for k in range(m)]
import itertools
deep = 3
final = []
mylist = [MM[i] for i in range(0,7-1)]
b = [str(i)+str(j)+str(k) for i in range(m) for j in range(m) for k in range(m)]
def DFS(b, deep, maxx, sourceoperators, path):
 initlist = []
 if deep > 0:
  print("deep=", deep)
  for aa,bb in itertools.combinations(sourceoperators, 2):
   print(aa,bb)
   if deep == maxx:
    finalresult = []
    op1xy = [aa[b[i][0:1]+b[i][1:2]] for i in range(len(b))]
    op1yz = [aa[b[i][1:2]+b[i][2:3]] for i in range(len(b))]
    op1xz = [aa[b[i][0:1]+b[i][2:3]] for i in range(len(b))]
    op2xy = [bb[b[i][0:1]+b[i][1:2]] for i in range(len(b))]
    op2yz = [bb[b[i][1:2]+b[i][2:3]] for i in range(len(b))]
    op2xz = [bb[b[i][0:1]+b[i][2:3]] for i in range(len(b))]
    if sum(op1xy) == 54:
      path.append([(deep, aa, "xy")])
    if sum(op1yz) == 54:
      path.append([(deep, aa, "yz")])
    if sum(op1xz) == 54:
      path.append([(deep, aa, "xz")])
    if sum(op2xy) == 54:
      path.append([(deep, bb, "xy")])
    if sum(op2yz) == 54:
      path.append([(deep, bb, "yz")])
    if sum(op2xz) == 54:      
      path.append([(deep, bb, "xz")])
    initlist.append(op1xy)
    initlist.append(op1yz)
    initlist.append(op1xz)
    initlist.append(op2xy)
    initlist.append(op2yz)
    initlist.append(op2xz)
   else:
    level = []
    for j in range(len(b)):
     op1xy = [aa[b[j][i]] for i in range(len(b[j]))]
     op2xy = [bb[b[j][i]] for i in range(len(b[j]))]
     if sum(op1xy) == 54:
      path.append([(deep, aa, "xy")])
     if sum(op2xy) == 54:
      path.append([(deep, bb, "xy")])
     level.append(op1xy)
     level.append(op2xy)
     initlist.append(op1xy)
     initlist.append(op2xy)
 if deep == maxx:
  b = []
  #print(len(list(itertools.combinations(initlist, 2))))
  for aaa,bbb in itertools.combinations(initlist, 2):     
   b.append([str(i)+str(j) for i,j in zip(aaa, bbb)])
  path = DFS(b, deep-1, maxx, sourceoperators, path)
 else:
  #print(len(list(itertools.combinations(initlist, 2))))
  for aaa,bbb in itertools.combinations(initlist, 2):
   b.append([str(i)+str(j) for i,j in zip(aaa, bbb)])
  path = DFS(b, deep-1, maxx, sourceoperators, path)
 return path

path = []
mresult = DFS(b, 2, 2, mylist, path)




Michael Selik於 2016年6月10日星期五 UTC+8上午6時45分14秒寫道：
> I like using Yelp's mrjob module (https://github.com/Yelp/mrjob) to run
> Python on Hadoop.
> 
> On Thu, Jun 9, 2016 at 2:56 AM Ho Yeung Lee <davidbenny2000@gmail.com>
> wrote:
> 
> > [... a bunch of code ...]
> 
> 
> If you want to describe a map-reduce problem, start with the data. What
> does a record of your input data look like?
> 
> Then think about your mapper. What key-value pairs will you extract from
> each line of data?
> 
> Then think about your reducer. For a single key and its associated values,
> what will you calculate?

[toc] | [prev] | [next] | [standalone]

#109787

On Thu, Jun 9, 2016, 9:11 PM Ho Yeung Lee <davidbenny2000@gmail.com> wrote:

> input are these six operators, output is finding full column of 27
> elements add together is 54
>
> i got memory error when searching this,
>
> i have difficulty in understanding map reduce and transforming my program
> into map reduce problem
>

Why do you think you need map-reduce?

You'll need to explain the problem in more detail. Instead of talking about
operators and columns, what is the actual, real-world problem?

>

[toc] | [prev] | [next] | [standalone]

#109817

there are six operator, and a logic table initial in b variable 

aa[b[j][i]] 
[aa[b[i][0:1]+b[i][2:3] 

are just like vlookup to find output of each operator 
acting on first column and second column, second column and third column 
, first column and third column 

and searching a output columns which is result sum is 27*2 
and record the path if succeed, 
it record the path  and output any path which has result is 27*2 described before 
op1(op2(op3(op1(op2(),),op1(op2(),))), op1(op2(),)) 

there are two cases, first cases b are logic table 
later case are for six operators acting on the result column from previous result before recursive call 



def DFS(b, deep, maxx, sourceoperators, path): 
    initlist = [] 
    if deep > 0: 
        print("deep=", deep) 
        for aa,bb in itertools.combinations(sourceoperators, 2): 
            print(aa,bb) 
            if deep == maxx: 
                finalresult = [] 
                op1xy = [aa[b[i][0:1]+b[i][1:2]] for i in range(len(b))] 
                op1yz = [aa[b[i][1:2]+b[i][2:3]] for i in range(len(b))] 
                op1xz = [aa[b[i][0:1]+b[i][2:3]] for i in range(len(b))] 
                op2xy = [bb[b[i][0:1]+b[i][1:2]] for i in range(len(b))] 
                op2yz = [bb[b[i][1:2]+b[i][2:3]] for i in range(len(b))] 
                op2xz = [bb[b[i][0:1]+b[i][2:3]] for i in range(len(b))] 
                if sum(op1xy) == 54: 
                    path.append([(deep, aa, b, "xy")]) 
                else: 
                    initlist.append(op1xy) 
                if sum(op1yz) == 54: 
                    path.append([(deep, aa, b, "yz")]) 
                else: 
                    initlist.append(op1yz) 
                if sum(op1xz) == 54: 
                    path.append([(deep, aa, b, "xz")]) 
                else: 
                    initlist.append(op1xz) 
                if sum(op2xy) == 54: 
                    path.append([(deep, bb, b, "xy")]) 
                else: 
                    initlist.append(op2xy) 
                if sum(op2yz) == 54: 
                    path.append([(deep, bb, b, "yz")]) 
                else: 
                    initlist.append(op2yz) 
                if sum(op2xz) == 54: 
                    path.append([(deep, bb, b, "xz")]) 
                else: 
                    initlist.append(op2xz) 
            else: 
                level = [] 
                for j in range(len(b)): 
                    op1xy = [aa[b[j][i]] for i in range(len(b[j]))] 
                    op2xy = [bb[b[j][i]] for i in range(len(b[j]))] 
                    if sum(op1xy) == 54: 
                        path.append([(deep, aa, b[j], "xy")]) 
                    else: 
                        initlist.append(op1xy) 
                    if sum(op2xy) == 54: 
                        path.append([(deep, bb, b[j], "xy")]) 
                    else: 
                        initlist.append(op2xy) 
                    level.extend([op1xy, op2xy]) 
    if deep == maxx: 
        b = [] 
    print("initlist=") 
    print(len(initlist)) 
    for aaa,bbb in itertools.combinations(initlist, 2): 
        b.append([str(i)+str(j) for i,j in zip(aaa, bbb)]) 
    if deep > 0: 
        path2 = DFS(b, deep-1, maxx, sourceoperators, path) 
        path.append(path2) 
    return path 

path = [] 
mresult = DFS(b, 2, 2, mylist, path) 


On Friday, June 10, 2016 at 10:04:09 PM UTC+8, Michael Selik wrote:
> On Thu, Jun 9, 2016, 9:11 PM Ho Yeung Lee <davidbenny2000@gmail.com> wrote:
> 
> > input are these six operators, output is finding full column of 27
> > elements add together is 54
> >
> > i got memory error when searching this,
> >
> > i have difficulty in understanding map reduce and transforming my program
> > into map reduce problem
> >
> 
> Why do you think you need map-reduce?
> 
> You'll need to explain the problem in more detail. Instead of talking about
> operators and columns, what is the actual, real-world problem?
> 
> >

[toc] | [prev] | [next] | [standalone]

#109821

>
> On Friday, June 10, 2016 at 10:04:09 PM UTC+8, Michael Selik wrote:
> >  You'll need to explain the problem in more detail. Instead of talking
> about operators and columns, what is the actual, real-world problem?


On Sat, Jun 11, 2016 at 2:21 AM meInvent bbird <jobmattcon@gmail.com> wrote:

> there are six operator, and a logic table initial in b variable
>

I wasn't asking about the algorithm. I was asking what the task is. If you
explain your research question that will help me (and I assume, the rest of
the mailing list) understand your problem better. Maybe we can suggest a
better algorithm. Give us some context: What data is being input? What is
the desired output? What decision will be made based on this calculation?

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web