Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #110020 > unrolled thread

is there a concurrent list for append in parallel programming in python? how to pass parameter in this parallel program with pool?

Back to article view | Back to comp.lang.python

Contents

  is there a concurrent list for append in parallel programming in python? how to pass parameter in this parallel program with pool? meInvent bbird <jobmattcon@gmail.com> - 2016-06-16 00:28 -0700
    Re: is there a concurrent list for append in parallel programming in python? how to pass parameter in this parallel program with pool? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2016-06-16 18:30 +1000
      Re: is there a concurrent list for append in parallel programming in python? how to pass parameter in this parallel program with pool? meInvent bbird <jobmattcon@gmail.com> - 2016-06-16 01:45 -0700
        Re: is there a concurrent list for append in parallel programming in python? how to pass parameter in this parallel program with pool? Ian Kelly <ian.g.kelly@gmail.com> - 2016-06-21 02:01 -0600
      Re: is there a concurrent list for append in parallel programming in python? how to pass parameter in this parallel program with pool? meInvent bbird <jobmattcon@gmail.com> - 2016-06-16 01:47 -0700

#110020 — is there a concurrent list for append in parallel programming in python? how to pass parameter in this parallel program with pool?

is there like c# have concurrent list ? 

i find something these, but how can it pass an initlist list variable 

is it doing the same function as itertools.combinations ? 

def comb(n, initlist): # the argument n is the number of items to select 
    res = list(itertools.combinations(initlist, n)) # create a list from the iterator 
    return res 

#p = Pool(8) 
#times = range(0, len(initlist)+1)                                                 
#values = p.map(comb, times) # pass the range as the sequence of arguments! 
#p.close() 
#p.join() 

[toc] | [next] | [standalone]

#110023

On Thursday 16 June 2016 17:28, meInvent bbird wrote:

> is there like c# have concurrent list ?

What is a concurrent list?

Can you link to the C# documentation for this?

To me, "concurrent" describes a style of execution flow, and "list" describes a 
data structure. I am struggling to understand what "concurrent list" means.

> i find something these, but how can it pass an initlist list variable

initlist = ['a', 'b', 'c']
result = comb(n, initlist)  # pass initlist


> is it doing the same function as itertools.combinations ?

It is calling itertools.combinations. So, yes, it is doing the same function as 
itertools.combinations.


> def comb(n, initlist): # the argument n is the number of items to select
>     res = list(itertools.combinations(initlist, n)) # create a list from the
>                                                     # iterator 
>     return res

This does not generate the combinations in parallel. It generates them one at a 
time, and then creates a list of them.

This is an interesting question. Somebody could probably write a parallel 
version of combinations. But I don't know that it would be very useful -- the 
limiting factor on combinations is more likely to be memory, not time.

Suppose you generate combinations of 100 items, taken 10 at a time. If I 
remember the formula for combinations correctly, the total number of 
combinations is:

100!/(10! * 90!) = 17310309456440

combinations in total. If each generated combination took just *one* byte of 
memory, that would require over 17 TB of RAM.



-- 
Steve

[toc] | [prev] | [next] | [standalone]

#110026

the name in c# is not called concurrent list, it is called
blockingcollection

dictionary called concurrent dictionary

thread safe these kind of things

https://msdn.microsoft.com/en-us/library/dd267312(v=vs.110).aspx

https://msdn.microsoft.com/en-us/library/dd997369(v=vs.110).aspx

https://msdn.microsoft.com/en-us/library/dd997305(v=vs.110).aspx


On Thursday, June 16, 2016 at 4:30:33 PM UTC+8, Steven D'Aprano wrote:
> On Thursday 16 June 2016 17:28, meInvent bbird wrote:
> 
> > is there like c# have concurrent list ?
> 
> What is a concurrent list?
> 
> Can you link to the C# documentation for this?
> 
> To me, "concurrent" describes a style of execution flow, and "list" describes a 
> data structure. I am struggling to understand what "concurrent list" means.
> 
> > i find something these, but how can it pass an initlist list variable
> 
> initlist = ['a', 'b', 'c']
> result = comb(n, initlist)  # pass initlist
> 
> 
> > is it doing the same function as itertools.combinations ?
> 
> It is calling itertools.combinations. So, yes, it is doing the same function as 
> itertools.combinations.
> 
> 
> > def comb(n, initlist): # the argument n is the number of items to select
> >     res = list(itertools.combinations(initlist, n)) # create a list from the
> >                                                     # iterator 
> >     return res
> 
> This does not generate the combinations in parallel. It generates them one at a 
> time, and then creates a list of them.
> 
> This is an interesting question. Somebody could probably write a parallel 
> version of combinations. But I don't know that it would be very useful -- the 
> limiting factor on combinations is more likely to be memory, not time.
> 
> Suppose you generate combinations of 100 items, taken 10 at a time. If I 
> remember the formula for combinations correctly, the total number of 
> combinations is:
> 
> 100!/(10! * 90!) = 17310309456440
> 
> combinations in total. If each generated combination took just *one* byte of 
> memory, that would require over 17 TB of RAM.
> 
> 
> 
> -- 
> Steve

[toc] | [prev] | [next] | [standalone]

#110219

On Thu, Jun 16, 2016 at 2:45 AM, meInvent bbird <jobmattcon@gmail.com> wrote:
> the name in c# is not called concurrent list, it is called
> blockingcollection
>
> dictionary called concurrent dictionary
>
> thread safe these kind of things
>
> https://msdn.microsoft.com/en-us/library/dd267312(v=vs.110).aspx
>
> https://msdn.microsoft.com/en-us/library/dd997369(v=vs.110).aspx
>
> https://msdn.microsoft.com/en-us/library/dd997305(v=vs.110).aspx

It sounds to me like you're looking for the Queue class:
https://docs.python.org/3/library/queue.html

Or possibly you want its multiprocessing equivalent, since it's not
clear to me which kind of concurrency you're interested in:
https://docs.python.org/3/library/multiprocessing.html#multiprocessing.Queue

Also, please don't top-post on this mailing list. Trimming quotations
and interleaving your replies is the accepted posting style here.

[toc] | [prev] | [next] | [standalone]

#110027

how can list be synchronized when multiprocessor working in it?

will one thread updating non-updated version, but another processor updating the version?

On Thursday, June 16, 2016 at 4:30:33 PM UTC+8, Steven D'Aprano wrote:
> On Thursday 16 June 2016 17:28, meInvent bbird wrote:
> 
> > is there like c# have concurrent list ?
> 
> What is a concurrent list?
> 
> Can you link to the C# documentation for this?
> 
> To me, "concurrent" describes a style of execution flow, and "list" describes a 
> data structure. I am struggling to understand what "concurrent list" means.
> 
> > i find something these, but how can it pass an initlist list variable
> 
> initlist = ['a', 'b', 'c']
> result = comb(n, initlist)  # pass initlist
> 
> 
> > is it doing the same function as itertools.combinations ?
> 
> It is calling itertools.combinations. So, yes, it is doing the same function as 
> itertools.combinations.
> 
> 
> > def comb(n, initlist): # the argument n is the number of items to select
> >     res = list(itertools.combinations(initlist, n)) # create a list from the
> >                                                     # iterator 
> >     return res
> 
> This does not generate the combinations in parallel. It generates them one at a 
> time, and then creates a list of them.
> 
> This is an interesting question. Somebody could probably write a parallel 
> version of combinations. But I don't know that it would be very useful -- the 
> limiting factor on combinations is more likely to be memory, not time.
> 
> Suppose you generate combinations of 100 items, taken 10 at a time. If I 
> remember the formula for combinations correctly, the total number of 
> combinations is:
> 
> 100!/(10! * 90!) = 17310309456440
> 
> combinations in total. If each generated combination took just *one* byte of 
> memory, that would require over 17 TB of RAM.
> 
> 
> 
> -- 
> Steve

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web