Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #78685 > unrolled thread

Issue in printing top 20 dictionary items by dictionary value

Back to article view | Back to comp.lang.python

Contents

  Issue in printing top 20 dictionary items by dictionary value Shiva <shivaji_tn@yahoo.com> - 2014-10-04 09:11 +0000
    Re: Issue in printing top 20 dictionary items by dictionary value Alexander Blinne <news@blinne.net> - 2014-10-04 12:28 +0200
      Re: Issue in printing top 20 dictionary items by dictionary value Shiva <shivaji_tn@yahoo.com> - 2014-10-04 17:36 +0000
        Re: Issue in printing top 20 dictionary items by dictionary value Larry Hudson <orgnut@yahoo.com> - 2014-10-04 14:44 -0700
          Re: Issue in printing top 20 dictionary items by dictionary value Shiva <shivaji_tn@yahoo.com> - 2014-10-05 06:58 +0000
      Re: Issue in printing top 20 dictionary items by dictionary value MRAB <python@mrabarnett.plus.com> - 2014-10-04 20:40 +0100
    Re: Issue in printing top 20 dictionary items by dictionary value Denis McMahon <denismfmcmahon@gmail.com> - 2014-10-05 01:11 +0000
      Re: Issue in printing top 20 dictionary items by dictionary value MRAB <python@mrabarnett.plus.com> - 2014-10-05 02:41 +0100

#78685 — Issue in printing top 20 dictionary items by dictionary value

Hi All,

I have written a function that 
-reads a file 
-splits the words and stores it in a dictionary as word(key) and the total
count of word in file (value).

I want to print the words with top 20 occurrences in the file in reverse
order - but can't figure it out. Here is my function:

def print_top(filename):

    #Open a file
    path = '/home/BCA/Documents/LearnPython/ic/'
    fname = path + filename
    print ('filename: ',fname)
    filetext = open(fname)

    #Read the file
    textstorage={}

    #print(type(textstorage))
    readall = filetext.read().lower()
    eachword = set(readall.split())

    #store split words as keys in dictionary
    for w in eachword:
        textstorage[w] = readall.count(w)

    #print top 20 items in dictionary by decending order of val
    # This bit is what I can't figure out.

    for dkey in (textstorage.keys()):
            print(dkey,sorted(textstorage[dkey]))??
           

[toc] | [next] | [standalone]

#78688

Am 04.10.2014 um 11:11 schrieb Shiva:
> Hi All,
> 
> I have written a function that 
> -reads a file 
> -splits the words and stores it in a dictionary as word(key) and the total
> count of word in file (value).
> 
> I want to print the words with top 20 occurrences in the file in reverse
> order - but can't figure it out.

As python is a high-level language with a comprehensive standard
library, there ought to be a built-in method to sort stuff... Wait, but
there ist:

sorted(iterable[, key][, reverse])

The tricky part is: a dictionary is not storing the order of its
entries. But you could just go with the list of your keys, sorted by the
count of your words and then go from there.

> Here is my function:
> 
> def print_top(filename):
> 
>     #Open a file
>     path = '/home/BCA/Documents/LearnPython/ic/'
>     fname = path + filename
>     print ('filename: ',fname)
>     filetext = open(fname)
> 
>     #Read the file
>     textstorage={}
> 
>     #print(type(textstorage))
>     readall = filetext.read().lower()
>     eachword = set(readall.split())
> 
>     #store split words as keys in dictionary
>     for w in eachword:
>         textstorage[w] = readall.count(w)
> 
>     #print top 20 items in dictionary by decending order of val
>     # This bit is what I can't figure out.

orderedwords = sorted(textstorage.keys(), key=textstorage.get, reverse=True)
toptwenty = orderedwords[:20]

for dkey in toptwenty:
    print(dkey,textstorage[dkey])

The interesting part is the "key=textstorage.get"-portion. This defines
according to what "key" two values of the iterable (the keys of the
dict) should be compared. The method textstorage.get will accept a word
and return the count of that word. This count is used to sort the words
- in reverse order.


Another remark: Your algorithm works, but imagine this: Your text could
be more than a few lines long and contain more than a few different
words. When you convert your text into the set of alle the words you
have to walk through it once. You only type one line into python, but
this is what python has to do at that point. Afterwards, for every
single word, you use the count-method of the complete text. This also
walks through the whole text - each time. So if your text has N
different words you walk through it N+2 times (your readall line also
walks completely through it!). It is possible to solve your problem
while only walking through the text only a few (independent of N) times
or even only a single time! And the larger your text gets, the more
important this gets. This is not an issue in python but in all of
computer science.

So try and rethink your algorithm with that goal in mind: Only walk
through the text once. Python makes strong use on iterators, also file
objects can be used as iterators. The count in your dictionary can be
updated while you walk through the text. The get-method has a keyword
argument for creating default values, which might be useful here.
Another thing worth of mentioning is, that python has exactly this kind
of machinery already built-in (collections.Counter), but you should try
and implement a simple version of it yourself as exercise.

Alexander

[toc] | [prev] | [next] | [standalone]

#78699

It works :
orderedwords = sorted(textstorage.keys(), key=textstorage.get)

The method textstorage.get will accept a word and return it's value which in
this instance is the count.

What I don't understand is:

    for w in eachword:
        textstorage[w]=textstorage.get(w, 0) + 1

How does textstorage.get(w,0)+1 give the count of the word??

Thanks,
Shiva

[toc] | [prev] | [next] | [standalone]

#78707

On 10/04/2014 10:36 AM, Shiva wrote:
>
> What I don't understand is:
>
>      for w in eachword:
>          textstorage[w]=textstorage.get(w, 0) + 1
>
> How does textstorage.get(w,0)+1 give the count of the word??
>

Very long-winded explanation:  (But to shorten it a bit, I'm going to use ts in place of 
textstorage.  Also lhs = left-hand-side and rhs = right-hand-side.)

What we're trying to do here is to update the word count.  We could (erroneously) write it as:

ts[w] = ts[w] + 1

If w already exists in the ts dictionary, this works fine.  But if it does not it will abort 
with a KeyError when it comes to the ts[w] on the rhs of the assignment.

The get() method is an alternate way of accessing the value of a key in a dictionary, but with a 
default value given as well.  Now let's break down the statement

ts[w] = ts.get(w, 0) + 1

Case 1:  w already exists in the ts dictionary:

ts.get(w, 0) gets the value of ts[w] (the current word count), adds 1, which is then used to 
update the word-count value of ts[w] (on the lhs of the assignment).

case2:  w does not exist in the ts dictionary:

ts.get(w, 0) gives the default value of 0, and 1 is added to that.  ts[w] on the lhs of the 
assignment does not exist, so a new entry is created in the ts dictionary with the given w as 
the key, and the value is initialized with the 1 from the get()+1.

Make sense?

      -=- Larry -=-

[toc] | [prev] | [next] | [standalone]

#78713

Larry Hudson <orgnut <at> yahoo.com.dmarc.invalid> writes:

> 
> On 10/04/2014 10:36 AM, Shiva wrote:
> >
> > What I don't understand is:
> >
> >      for w in eachword:
> >          textstorage[w]=textstorage.get(w, 0) + 1
> >
> > How does textstorage.get(w,0)+1 give the count of the word??
> >
> 
> Very long-winded explanation:  (But to shorten it a bit, I'm going to use
ts in place of 
> textstorage.  Also lhs = left-hand-side and rhs = right-hand-side.)
> 
> What we're trying to do here is to update the word count.  We could
(erroneously) write it as:
> 
> ts[w] = ts[w] + 1
> 
> If w already exists in the ts dictionary, this works fine.  But if it does
not it will abort 
> with a KeyError when it comes to the ts[w] on the rhs of the assignment.
> 
> The get() method is an alternate way of accessing the value of a key in a
dictionary, but with a 
> default value given as well.  Now let's break down the statement
> 
> ts[w] = ts.get(w, 0) + 1
> 
> Case 1:  w already exists in the ts dictionary:
> 
> ts.get(w, 0) gets the value of ts[w] (the current word count), adds 1,
which is then used to 
> update the word-count value of ts[w] (on the lhs of the assignment).
> 
> case2:  w does not exist in the ts dictionary:
> 
> ts.get(w, 0) gives the default value of 0, and 1 is added to that.  ts[w]
on the lhs of the 
> assignment does not exist, so a new entry is created in the ts dictionary
with the given w as 
> the key, and the value is initialized with the 1 from the get()+1.
> 
> Make sense?
> 
>       -=- Larry -=-
> 
> 

Hi Larry,

Thanks for the explanation - I was a bit confused as get() operation in this
case would have got None for words occurring the first time.
Now I understand by writing a small example in the interpreter:

>>> dt={}
>>> splitw=('aa','bb','cc')
>>> for w in splitw:
...     dt[w]=dt.get(w,0)
...     
>>> dt
{'cc': 0, 'bb': 0, 'aa': 0}

So we just increment 0 to 1 for count.

Thanks,
Pradeep

[toc] | [prev] | [next] | [standalone]

#78704

On 2014-10-04 18:36, Shiva wrote:
 > It works :
 > orderedwords = sorted(textstorage.keys(), key=textstorage.get)
 >
 > The method textstorage.get will accept a word and return it's value
 > which in this instance is the count.
 >
 > What I don't understand is:
 >
 >      for w in eachword:
 >          textstorage[w]=textstorage.get(w, 0) + 1
 >
 > How does textstorage.get(w,0)+1 give the count of the word??
 >
What textstorage.get(w, 0) says is: if the word is in the dict, return
its count, else return 0.

[toc] | [prev] | [next] | [standalone]

#78709

On Sat, 04 Oct 2014 09:11:43 +0000, Shiva wrote:

> I have written a function that -reads a file -splits the words and
> stores it in a dictionary as word(key) and the total count of word in
> file (value).
> 
> I want to print the words with top 20 occurrences in the file in reverse
> order - but can't figure it out. Here is my function:

Once you've generated your dictionary, use a list comprehension to turn 
it into a list of tuples of ( word, count ). Sort the list of tuples 
according to the count element. Select the top 20. copy it to a new list 
and reverse that list. now loop through the second list and print results.

-- 
Denis McMahon, denismfmcmahon@gmail.com

[toc] | [prev] | [next] | [standalone]

#78710

On 2014-10-05 02:11, Denis McMahon wrote:
> On Sat, 04 Oct 2014 09:11:43 +0000, Shiva wrote:
>
>> I have written a function that -reads a file -splits the words and
>> stores it in a dictionary as word(key) and the total count of word
>> in file (value).
>>
>> I want to print the words with top 20 occurrences in the file in
>> reverse order - but can't figure it out. Here is my function:
>
> Once you've generated your dictionary, use a list comprehension to
> turn it into a list of tuples of ( word, count ). Sort the list of
> tuples according to the count element. Select the top 20. copy it to
> a new list and reverse that list. now loop through the second list
> and print results.
>
You don't need a list comprehension to turn it into a list of tuples,
and you don't need to sort and reverse as separate steps.

In fact, there's a class in the collections module that'll do virtually
all of the work for you!

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web