Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #60748 > unrolled thread

how to implement a queue-like container with sort function

Back to article view | Back to comp.lang.python

Contents

  how to implement a queue-like container  with sort function iMath <redstone-cold@163.com> - 2013-11-28 17:54 -0800
    Re: how to implement a queue-like container with sort function Chris Angelico <rosuav@gmail.com> - 2013-11-29 13:03 +1100
      Re: how to implement a queue-like container with sort function Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-11-29 03:32 +0000
    Re: how to implement a queue-like container  with sort function iMath <redstone-cold@163.com> - 2013-11-28 18:04 -0800
      Re: how to implement a queue-like container with sort function Chris Angelico <rosuav@gmail.com> - 2013-11-29 13:06 +1100
        Re: how to implement a queue-like container with sort function iMath <redstone-cold@163.com> - 2013-11-28 18:32 -0800
      Re: how to implement a queue-like container  with sort function Cameron Simpson <cs@zip.com.au> - 2013-11-29 17:19 +1100
    Re: how to implement a queue-like container  with sort function MRAB <python@mrabarnett.plus.com> - 2013-11-29 02:23 +0000
    Re: how to implement a queue-like container  with sort function Terry Reedy <tjreedy@udel.edu> - 2013-11-28 21:31 -0500
    Re: how to implement a queue-like container  with sort function Gregory Ewing <greg.ewing@canterbury.ac.nz> - 2013-11-29 17:53 +1300
    Re: how to implement a queue-like container  with sort function iMath <redstone-cold@163.com> - 2013-11-29 04:33 -0800
      Re: how to implement a queue-like container  with sort function Mark Lawrence <breamoreboy@yahoo.co.uk> - 2013-11-29 14:57 +0000
        Re: how to implement a queue-like container  with sort function iMath <redstone-cold@163.com> - 2013-12-02 03:41 -0800
          Re: how to implement a queue-like container  with sort function Ned Batchelder <ned@nedbatchelder.com> - 2013-12-02 06:58 -0500
          Re: how to implement a queue-like container with sort function Chris Angelico <rosuav@gmail.com> - 2013-12-02 23:04 +1100
          Re: how to implement a queue-like container with sort function Ned Batchelder <ned@nedbatchelder.com> - 2013-12-02 07:26 -0500

#60748 — how to implement a queue-like container with sort function

I want to a fixed length list-like container, it should have a sorted()-like function that I can use to sort it,I think there should also a function I can use it to detect whether the numbers of items in it reaches the length of the container , because if the numbers of items in it reaches the length(fixed) of the container,I want to process the data in it .Is there a container in Python like this ?If not, what base container should be used to implement such container?

the container is similar to queue ,but queue doesn't have a sort  function 

[toc] | [next] | [standalone]

#60749 — Re: how to implement a queue-like container with sort function

On Fri, Nov 29, 2013 at 12:54 PM, iMath <redstone-cold@163.com> wrote:
> the container is similar to queue ,but queue doesn't have a sort  function

It's either a queue that can be sorted, or a list with a length limit.
You could fairly easily implement either, because in Python, anything
can be subclassed. But I think possibly the easiest way is to simply
turn your queue into a list when you want to work with it:

>>> import queue
>>> a=queue.Queue(10)
>>> a.put("asdf")
>>> a.put("qwer")
>>> a.put("zxcv")
>>> a.put("1234")
>>> sorted(a.queue)
['1234', 'asdf', 'qwer', 'zxcv']

Though I don't know if this is what you meant. Putting too much onto a
queue.Queue will block. Were you wanting it, instead, to discard
entries? If so, which?

ChrisA

[toc] | [prev] | [next] | [standalone]

#60756 — Re: how to implement a queue-like container with sort function

On Fri, 29 Nov 2013 13:03:00 +1100, Chris Angelico wrote:

> On Fri, Nov 29, 2013 at 12:54 PM, iMath <redstone-cold@163.com> wrote:
>> the container is similar to queue ,but queue doesn't have a sort 
>> function
> 
> It's either a queue that can be sorted, or a list with a length limit.
> You could fairly easily implement either, because in Python, anything
> can be subclassed. But I think possibly the easiest way is to simply
> turn your queue into a list when you want to work with it:
> 
>>>> import queue
>>>> a=queue.Queue(10)
>>>> a.put("asdf")
>>>> a.put("qwer")
>>>> a.put("zxcv")
>>>> a.put("1234")
>>>> sorted(a.queue)
> ['1234', 'asdf', 'qwer', 'zxcv']
> 
> Though I don't know if this is what you meant. Putting too much onto a
> queue.Queue will block. Were you wanting it, instead, to discard
> entries? If so, which?


Unless the OP needs all the extra threading-related powers of the queue 
module, I'd just stick to a simple, single-threaded queue object. These 
easiest way to do that is with a deque, which is a double-ended queue. 
The threading Queue class is based on a deque, so this will be at least 
as fast:

py> from collections import deque
py> my_queue = deque([], 5)  # maximum of five items
py> my_queue.append(7)
py> my_queue.append(3)
py> my_queue.append(9)
py> sorted(my_queue)
[3, 7, 9]
py> my_queue.append(2)
py> my_queue.append(0)
py> my_queue.append(4)
py> my_queue
deque([3, 9, 2, 0, 4], maxlen=5)
py> sorted(my_queue)
[0, 2, 3, 4, 9]



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#60750

All in all,I want to first fill the container, then sort it and process all the contents in it 

[toc] | [prev] | [next] | [standalone]

#60751 — Re: how to implement a queue-like container with sort function

On Fri, Nov 29, 2013 at 1:04 PM, iMath <redstone-cold@163.com> wrote:
> All in all,I want to first fill the container, then sort it and process all the contents in it

Where does the length limit come in?

By the way, a little context helps a lot with following a thread.

ChrisA

[toc] | [prev] | [next] | [standalone]

#60753 — Re: how to implement a queue-like container with sort function

hey , you used
>>> sorted(a.queue)

this means the queue.Queue() has an attribute queue ,but I cannot find it described in the DOC ,where you find it ?

[toc] | [prev] | [next] | [standalone]

#60759

On 28Nov2013 18:04, iMath <redstone-cold@163.com> wrote:
> All in all,I want to first fill the container, then sort it and process all the contents in it 

Automatically?

It sounds like an I/O buffer, in the sense that a block buffered
output stream does a write on buffer full.

Something like this:

  class Chunkifier:

    def __init__(self, size, process):
      self.size = size
      self.buffer = []

    def append(self, item):
      self.buffer.append(item)
      if len(self.buffer) >= self.size:
        process(sorted(self.buffer))
        self.buffer = []

    def extend(self, items):
      for item in items:
        self.append(item)

  def burp(items):
    for item in items:
      print item

  burper = Chunkifier(10, burp)
  burper.extend( (9,8,7,6,5,4,3,2,1,2,3,4,5,6,7,8,9,8,7,6,5,4,3,2,1) )

Totally untested, buyer beware, worth what you paid for it, etc...

Cheers,
-- 
Cameron Simpson <cs@zip.com.au>

It is necessary for technical reasons that these warheads be stored with
the top at the bottom and the bottom at the top. In order that there may
be no doubt as to which is the top and which is the bottom, for storage
purposes it will be seen that the bottom of each head has been labelled
with the word TOP.      - Instructions for storing British nuclear warheads

[toc] | [prev] | [next] | [standalone]

#60752

On 29/11/2013 01:54, iMath wrote:
> I want to a fixed length list-like container, it should have a
> sorted()-like function that I can use to sort it,I think there should
> also a function I can use it to detect whether the numbers of items
> in it reaches the length of the container , because if the numbers of
> items in it reaches the length(fixed) of the container,I want to
> process the data in it .Is there a container in Python like this ?If
> not, what base container should be used to implement such container?
>
> the container is similar to queue ,but queue doesn't have a sort
> function
>
This is Python. You don't have to base it on an existing container.

Write a list of the methods it should have. If none of the existing
classes seem suitable as a superclass, then don't bother, just write it
from scratch. You can always use an existing container, such as a list,
as part of its implementation.

[toc] | [prev] | [next] | [standalone]

#60754

On 11/28/2013 8:54 PM, iMath wrote:
> I want to a fixed length list-like container, it should have a sorted()-like function that I can use to sort it,I think there should also a function I can use it to detect whether the numbers of items in it reaches the length of the container , because if the numbers of items in it reaches the length(fixed) of the container,I want to process the data in it .Is there a container in Python like this ?If not, what base container should be used to implement such container?
>
> the container is similar to queue ,but queue doesn't have a sort function

For single thread use, subclass list, add a new .put method that checks 
the list size and either does self.append or self.sort(); process(list).

For multiple threads, you need the queue module, which has extra 
features (and baggage). A PriorityQueue yields items in sorted order. 
Subclass it and override .put to either call the original .put or start 
a process thread.

-- 
Terry Jan Reedy

[toc] | [prev] | [next] | [standalone]

#60758

iMath wrote:
> the container is similar to queue ,but queue doesn't have a sort  function 

You can use a list as a queue. If you have a list l, then
l.append(x) will add an item to the end, and l.pop(0) will
remove the first item and return it.

Then you just need to check the length of the list before
adding an item, and if it's full, do something to process
the items first.

You can encapsulate all this inside a class if you want,
but that's optional.

-- 
Greg

[toc] | [prev] | [next] | [standalone]

#60763

it seems  PriorityQueue satisfy my requirement here .

BTW ,the Queue object has an attribute 'queue' ,but I cannot find it described in the DOC ,what it means ?

[toc] | [prev] | [next] | [standalone]

#60767

On 29/11/2013 12:33, iMath wrote:
>
> BTW ,the Queue object has an attribute 'queue' ,but I cannot find it described in the DOC ,what it means ?
>

Really? AttributeError: type object 'Queue' has no attribute 'queue'

-- 
Python is the second best programming language in the world.
But the best has yet to be invented.  Christian Tismer

Mark Lawrence

[toc] | [prev] | [next] | [standalone]

#60859

在 2013年11月29日星期五UTC+8下午10时57分36秒，Mark Lawrence写道：
> On 29/11/2013 12:33, iMath wrote:
> 
> >
> 
> > BTW ,the Queue object has an attribute 'queue' ,but I cannot find it described in the DOC ,what it means ?
> 
> >
> 
> 
> 
> Really? AttributeError: type object 'Queue' has no attribute 'queue'
> 
> 
> 
> -- 
> 
> Python is the second best programming language in the world.
> 
> But the best has yet to be invented.  Christian Tismer
> 
> 
> 
> Mark Lawrence

you can do a check by hasattr()

[toc] | [prev] | [next] | [standalone]

#60861

On 12/2/13 6:41 AM, iMath wrote:
> 在 2013年11月29日星期五UTC+8下午10时57分36秒，Mark Lawrence写道：
>> On 29/11/2013 12:33, iMath wrote:
>>
>>>
>>
>>> BTW ,the Queue object has an attribute 'queue' ,but I cannot find it described in the DOC ,what it means ?
>>
>>>
>>
>>
>>
>> Really? AttributeError: type object 'Queue' has no attribute 'queue'
>>
>>
>>
>> --
>>
>> Python is the second best programming language in the world.
>>
>> But the best has yet to be invented.  Christian Tismer
>>
>>
>>
>> Mark Lawrence
>
> you can do a check by hasattr()
>

Yes, a Queue object has a queue attribute:

     >>> import Queue
     >>> q = Queue.Queue()
     >>> q.queue
     deque([])

But you shouldn't use it.  It's part of the implementation of Queue, not 
meant for you to use directly.  In particular, if you use it directly, 
you are skipping all synchronization, which is the main reason to use a 
Queue in the first place.

It should have been named "_queue". We'll add that to the list of PEP-8 
violations in the Queue module! :)

--Ned.

[toc] | [prev] | [next] | [standalone]

#60862 — Re: how to implement a queue-like container with sort function

On Mon, Dec 2, 2013 at 10:58 PM, Ned Batchelder <ned@nedbatchelder.com> wrote:
> Yes, a Queue object has a queue attribute:
>
>     >>> import Queue
>     >>> q = Queue.Queue()
>     >>> q.queue
>     deque([])
>
> But you shouldn't use it.  It's part of the implementation of Queue, not
> meant for you to use directly.  In particular, if you use it directly, you
> are skipping all synchronization, which is the main reason to use a Queue in
> the first place.

I should apologize here; when the OP said "queue", I immediately
noticed that I could import that and use it, and mistakenly started my
testing on that, instead of using the deque type. It's deque that
should be used here. Queue is just a wrapper around deque that adds
functionality that has nothing to do with what's needed here.

ChrisA

[toc] | [prev] | [next] | [standalone]

#60863 — Re: how to implement a queue-like container with sort function

On 12/2/13 7:04 AM, Chris Angelico wrote:
> On Mon, Dec 2, 2013 at 10:58 PM, Ned Batchelder <ned@nedbatchelder.com> wrote:
>> Yes, a Queue object has a queue attribute:
>>
>>      >>> import Queue
>>      >>> q = Queue.Queue()
>>      >>> q.queue
>>      deque([])
>>
>> But you shouldn't use it.  It's part of the implementation of Queue, not
>> meant for you to use directly.  In particular, if you use it directly, you
>> are skipping all synchronization, which is the main reason to use a Queue in
>> the first place.
>
> I should apologize here; when the OP said "queue", I immediately
> noticed that I could import that and use it, and mistakenly started my
> testing on that, instead of using the deque type. It's deque that
> should be used here. Queue is just a wrapper around deque that adds
> functionality that has nothing to do with what's needed here.
>
> ChrisA
>

Actually, I had a long conversation in the #python IRC channel with the 
OP at the same time he was posting the question here, and it turns out 
he knows exactly how many entries are going into the "queue", so a 
plain-old list is the best solution.  I don't know quite where the idea 
of limiting the number of entries came from.

--Ned.

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web