Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #111965 > unrolled thread

Python text file fetch specific part of line

Back to article view | Back to comp.lang.python

Contents

  Python text file fetch specific part of line Arshpreet Singh <arsh840@gmail.com> - 2016-07-27 22:12 -0700
    Re: Python text file fetch specific part of line cs@zip.com.au - 2016-07-28 17:04 +1000
      Re: Python text file fetch specific part of line honeygne@gmail.com - 2016-08-01 23:55 -0700
    Re: Python text file fetch specific part of line Gordon Levi <gordon@address.invalid> - 2016-07-28 19:28 +1000
      Re: Python text file fetch specific part of line cs@zip.com.au - 2016-07-29 08:17 +1000
        Re: Python text file fetch specific part of line Gordon Levi <gordon@address.invalid> - 2016-07-29 18:42 +1000
          Re: Python text file fetch specific part of line cs@zip.com.au - 2016-07-30 12:46 +1000

#111965 — Python text file fetch specific part of line

I am writing Imdb scrapper, and getting available list of titles from IMDB website which provide txt file in very raw format, Here is the one part of file(http://pastebin.com/fpMgBAjc) as the file provides tags like Distribution  Votes,Rank,Title I want to parse title names, I tried with readlines() method but it returns only list which is quite heterogeneous, is it possible that I can parse each value comes under title section?

[toc] | [next] | [standalone]

#111969

On 27Jul2016 22:12, Arshpreet Singh <arsh840@gmail.com> wrote:
>I am writing Imdb scrapper, and getting available list of titles from IMDB 
>website which provide txt file in very raw format, Here is the one part of 
>file(http://pastebin.com/fpMgBAjc) as the file provides tags like Distribution  
>Votes,Rank,Title I want to parse title names, I tried with readlines() method 
>but it returns only list which is quite heterogeneous, is it possible that I 
>can parse each value comes under title section?

Just for etiquette: please just post text snippets like that inline in your 
text. Some people don't like fetching random URLs, and some of us are not 
always online when reading and replying to email. Either way, having the text 
in the message, especially when it is small, is preferable.

To your question:

Your sample text looks like this:

    New  Distribution  Votes  Rank  Title
      0000000125  1680661   9.2  The Shawshank Redemption (1994)
      0000000125  1149871   9.2  The Godfather (1972)
      0000000124  786433   9.0  The Godfather: Part II (1974)
      0000000124  1665643   8.9  The Dark Knight (2008)
      0000000133  860145   8.9  Schindler's List (1993)
      0000000133  444718   8.9  12 Angry Men (1957)
      0000000123  1317267   8.9  Pulp Fiction (1994)
      0000000124  1209275   8.9  The Lord of the Rings: The Return of the King 
(2003)
      0000000123  500803   8.9  Il buono, il brutto, il cattivo (1966)
      0000000133  1339500   8.8  Fight Club (1999)
      0000000123  1232468   8.8  The Lord of the Rings: The Fellowship of the 
Ring (2001)
      0000000223  832726   8.7  Star Wars: Episode V - The Empire Strikes Back 
(1980)
      0000000233  1243066   8.7  Forrest Gump (1994)
      0000000123  1459168   8.7  Inception (2010)
      0000000223  1094504   8.7  The Lord of the Rings: The Two Towers (2002)
      0000000232  676479   8.7  One Flew Over the Cuckoo's Nest (1975)
      0000000232  724590   8.7  Goodfellas (1990)
      0000000233  1211152   8.7  The Matrix (1999)

Firstly, I would suggest you not use readlines(), it pulls all the text into 
memory. For small text like this is it ok, but some things can be arbitrarily 
large, so it is something to avoid if convenient. Normally you can just iterate 
over a file and get lines.

You want "text under the Title." Looking at it, I would be inclined to say that 
the first line is a header and the rest consist of 4 columns: a number 
(distribution?), a vote count, a rank and the rest (title plus year).

You can parse data like that like this (untested):

  # presumes `fp` is reading from the text
  for n, line in enumerate(fp):
    if n == 0:
      # heading, skip it
      continue
    distnum, nvotes, rank, etc = split(line, 3)
    ... do stuff with the various fields ...

I hope that gets you going. If not, return with what code you have, what 
happened, and what you actually wanted to happen and we may help further.

Cheers,
Cameron Simpson <cs@zip.com.au>

[toc] | [prev] | [next] | [standalone]

#112206

On Thursday, July 28, 2016 at 1:00:17 PM UTC+5:30, c...@zip.com.au wrote:
> On 27Jul2016 22:12, Arshpreet Singh <arsh840@gmail.com> wrote:
> >I am writing Imdb scrapper, and getting available list of titles from IMDB 
> >website which provide txt file in very raw format, Here is the one part of 
> >file(http://pastebin.com/fpMgBAjc) as the file provides tags like Distribution  
> >Votes,Rank,Title I want to parse title names, I tried with readlines() method 
> >but it returns only list which is quite heterogeneous, is it possible that I 
> >can parse each value comes under title section?
> 
> Just for etiquette: please just post text snippets like that inline in your 
> text. Some people don't like fetching random URLs, and some of us are not 
> always online when reading and replying to email. Either way, having the text 
> in the message, especially when it is small, is preferable.
> 
> To your question:
> 
> Your sample text looks like this:
> 
>     New  Distribution  Votes  Rank  Title
>       0000000125  1680661   9.2  The Shawshank Redemption (1994)
>       0000000125  1149871   9.2  The Godfather (1972)
>       0000000124  786433   9.0  The Godfather: Part II (1974)
>       0000000124  1665643   8.9  The Dark Knight (2008)
>       0000000133  860145   8.9  Schindler's List (1993)
>       0000000133  444718   8.9  12 Angry Men (1957)
>       0000000123  1317267   8.9  Pulp Fiction (1994)
>       0000000124  1209275   8.9  The Lord of the Rings: The Return of the King 
> (2003)
>       0000000123  500803   8.9  Il buono, il brutto, il cattivo (1966)
>       0000000133  1339500   8.8  Fight Club (1999)
>       0000000123  1232468   8.8  The Lord of the Rings: The Fellowship of the 
> Ring (2001)
>       0000000223  832726   8.7  Star Wars: Episode V - The Empire Strikes Back 
> (1980)
>       0000000233  1243066   8.7  Forrest Gump (1994)
>       0000000123  1459168   8.7  Inception (2010)
>       0000000223  1094504   8.7  The Lord of the Rings: The Two Towers (2002)
>       0000000232  676479   8.7  One Flew Over the Cuckoo's Nest (1975)
>       0000000232  724590   8.7  Goodfellas (1990)
>       0000000233  1211152   8.7  The Matrix (1999)
> 
> Firstly, I would suggest you not use readlines(), it pulls all the text into 
> memory. For small text like this is it ok, but some things can be arbitrarily 
> large, so it is something to avoid if convenient. Normally you can just iterate 
> over a file and get lines.
> 
> You want "text under the Title." Looking at it, I would be inclined to say that 
> the first line is a header and the rest consist of 4 columns: a number 
> (distribution?), a vote count, a rank and the rest (title plus year).
> 
> You can parse data like that like this (untested):
> 
>   # presumes `fp` is reading from the text
>   for n, line in enumerate(fp):
>     if n == 0:
>       # heading, skip it
>       continue
>     distnum, nvotes, rank, etc = split(line, 3)
>     ... do stuff with the various fields ...
> 
> I hope that gets you going. If not, return with what code you have, what 
> happened, and what you actually wanted to happen and we may help further.
Thanks I am able to do it with following:
https://github.com/alberanid/imdbpy/blob/master/bin/imdbpy2sql.py (it was very helpful)

python imdbpy2sql.py -d <.txt files downloaded from IMDB> -u sqlite:/where/to/save/db --sqlite-transactions

[toc] | [prev] | [next] | [standalone]

#111975

Arshpreet Singh <arsh840@gmail.com> wrote:

>I am writing Imdb scrapper, and getting available list of titles from IMDB website which provide txt file in very raw format, Here is the one part of file(http://pastebin.com/fpMgBAjc) as the file provides tags like Distribution  Votes,Rank,Title I want to parse title names, I tried with readlines() method but it returns only list which is quite heterogeneous, is it possible that I can parse each value comes under title section?

Beautiful Soup will make your task much easier
<https://www.crummy.com/software/BeautifulSoup/>.

[toc] | [prev] | [next] | [standalone]

#111999

On 28Jul2016 19:28, Gordon Levi <gordon@address.invalid> wrote:
>Arshpreet Singh <arsh840@gmail.com> wrote:
>>I am writing Imdb scrapper, and getting available list of titles from IMDB 
>>website which provide txt file in very raw format, Here is the one part of 
>>file(http://pastebin.com/fpMgBAjc) as the file provides tags like 
>>Distribution  Votes,Rank,Title I want to parse title names, I tried with 
>>readlines() method but it returns only list which is quite heterogeneous, is 
>>it possible that I can parse each value comes under title section?
>
>Beautiful Soup will make your task much easier
><https://www.crummy.com/software/BeautifulSoup/>.

Did you look at his sample data? Plain text, not HTML or XML. Beautiful Soup is 
not what he needs here.

Cheers,
Cameron Simpson <cs@zip.com.au>

[toc] | [prev] | [next] | [standalone]

#112006

cs@zip.com.au wrote:

>On 28Jul2016 19:28, Gordon Levi <gordon@address.invalid> wrote:
>>Arshpreet Singh <arsh840@gmail.com> wrote:
>>>I am writing Imdb scrapper, and getting available list of titles from IMDB 
>>>website which provide txt file in very raw format, Here is the one part of 
>>>file(http://pastebin.com/fpMgBAjc) as the file provides tags like 
>>>Distribution  Votes,Rank,Title I want to parse title names, I tried with 
>>>readlines() method but it returns only list which is quite heterogeneous, is 
>>>it possible that I can parse each value comes under title section?
>>
>>Beautiful Soup will make your task much easier
>><https://www.crummy.com/software/BeautifulSoup/>.
>
>Did you look at his sample data?

No. I read he was "writing an IMDB scraper, and getting the available
list of titles from the IMDB web site". It's here
<http://www.imdb.com/>.  
> Plain text, not HTML or XML. Beautiful Soup is 
>not what he needs here.

Fortunately the OP told us his application rather than just telling us
his current problem. His life would be much easier if he ignored the
plain text he has obtained so far and started again using a Beautiful
Soup tutorial. 

[toc] | [prev] | [next] | [standalone]

#112038

On 29Jul2016 18:42, Gordon Levi <gordon@address.invalid> wrote:
>cs@zip.com.au wrote:
>
>>On 28Jul2016 19:28, Gordon Levi <gordon@address.invalid> wrote:
>>>Arshpreet Singh <arsh840@gmail.com> wrote:
>>>>I am writing Imdb scrapper, and getting available list of titles from IMDB
>>>>website which provide txt file in very raw format, Here is the one part of
>>>>file(http://pastebin.com/fpMgBAjc) as the file provides tags like
>>>>Distribution  Votes,Rank,Title I want to parse title names, I tried with
>>>>readlines() method but it returns only list which is quite heterogeneous, is
>>>>it possible that I can parse each value comes under title section?
>>>
>>>Beautiful Soup will make your task much easier
>>><https://www.crummy.com/software/BeautifulSoup/>.
>>
>>Did you look at his sample data?
>
>No. I read he was "writing an IMDB scraper, and getting the available
>list of titles from the IMDB web site". It's here
><http://www.imdb.com/>.
>
>> Plain text, not HTML or XML. Beautiful Soup is
>>not what he needs here.
>
>Fortunately the OP told us his application rather than just telling us
>his current problem. His life would be much easier if he ignored the
>plain text he has obtained so far and started again using a Beautiful
>Soup tutorial.

Or bypass IMDB's computer unfriendliness and go straight to http://omdbapi.com/

You can have JSON directly from it, and avoid BS entirely. BS is an amazing 
library, but is essentially a workaround for computer-hostile websites: those 
not providing clean machine readable data, and only unstable mutable HTML 
output.

Cheers,
Cameron Simpson <cs@zip.com.au>

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web