Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]


Groups > comp.soft-sys.math.mathematica > #13506 > unrolled thread

Obtaining Random LIne from A file

Started byRamiro Barrantes <rbarrant@uvm.edu>
First post2013-02-16 06:07 +0000
Last post2013-02-19 23:52 +0000
Articles 8 — 5 participants

Back to article view | Back to comp.soft-sys.math.mathematica


Contents

  Obtaining Random LIne from A file Ramiro Barrantes <rbarrant@uvm.edu> - 2013-02-16 06:07 +0000
    Re: Obtaining Random LIne from A file David Bailey <dave@removedbailey.co.uk> - 2013-02-17 09:08 +0000
      Re: Obtaining Random LIne from A file awnl <awnl@gmx-topmail.de> - 2013-02-18 10:59 +0000
        Re: Obtaining Random LIne from A file David Bailey <dave@removedbailey.co.uk> - 2013-02-19 06:09 +0000
      Re: Obtaining Random LIne from A file Ramiro <ramiro.barrantes@gmail.com> - 2013-02-19 06:09 +0000
        Re: Obtaining Random LIne from A file "Kevin J. McCann" <kjm@KevinMcCann.com> - 2013-02-19 23:51 +0000
        Re: Obtaining Random LIne from A file awnl <awnl@gmx-topmail.de> - 2013-02-19 23:52 +0000
        Re: Obtaining Random LIne from A file David Bailey <dave@removedbailey.co.uk> - 2013-02-19 23:52 +0000

#13506 — Obtaining Random LIne from A file

FromRamiro Barrantes <rbarrant@uvm.edu>
Date2013-02-16 06:07 +0000
SubjectObtaining Random LIne from A file
Message-ID<kfn7nt$qaj$1@smc.vnet.net>
Hello,

I would like to get a random line from a file, I know this can be done 
with Mathematica but I am playing with using sed to see if it goes 
faster, say I want to get line 1000

In mathematica it would be:

<<"! sed -n p1000 filename.txt"

However, I am trying to put the filename as a variable, say

filename="hugefile.txt"

cmd="! sed -n p1000 "<>filename
<<cmd

does not work.

How can I do this?

Lastly, I am getting a randomline using mathematica doing:

getRandomLine[file_, n_] :=
  Block[{i = RandomInteger[{1, n}], str = OpenRead[file], res},
   Skip[str, "String", i];
   res = Read[str, Expression];
   Close[str];
   res[[2]]
   ]

However, it is very slow so I was going to try with sed.Any suggestions?

Thanks in advance,
Ramiro

[toc] | [next] | [standalone]


#13515

FromDavid Bailey <dave@removedbailey.co.uk>
Date2013-02-17 09:08 +0000
Message-ID<kfq6mb$4us$1@smc.vnet.net>
In reply to#13506
On 16/02/2013 06:07, Ramiro Barrantes wrote:
> Hello,
>
> I would like to get a random line from a file, I know this can be done
> with Mathematica but I am playing with using sed to see if it goes
> faster, say I want to get line 1000
>
> In mathematica it would be:
>
> <<"! sed -n p1000 filename.txt"
>
> However, I am trying to put the filename as a variable, say
>
> filename="hugefile.txt"
>
> cmd="! sed -n p1000 "<>filename
> <<cmd
>
> does not work.
>
> How can I do this?
>
> Lastly, I am getting a randomline using mathematica doing:
>
> getRandomLine[file_, n_] :=
>    Block[{i = RandomInteger[{1, n}], str = OpenRead[file], res},
>     Skip[str, "String", i];
>     res = Read[str, Expression];
>     Close[str];
>     res[[2]]
>     ]
>
> However, it is very slow so I was going to try with sed.Any suggestions?
>
> Thanks in advance,
> Ramiro
>
>
I would stick with Mathematica to do this job! How big is the file 
(number of lines and number of bytes)? If it will fit inside Mathematica 
comfortable, I'd see how it works to read it all in as a list of strings 
and pick the one you want:

xx=ReadList["C:\\some file",String];//Timing

Then you have an array of strings, and you can select what you want 
directly.

Remember, the basic problem with reading at an arbitrary position in a 
text file, is that if the line lengths are not the same, any algorithm 
has to read every line before the one you want! If you create this file, 
you should consider packing the lines to make them all the same length - 
then you could access what you want very efficiently (but with a little 
more coding!)

David Bailey
http://www.dbaileyconsultancy.co.uk


[toc] | [prev] | [next] | [standalone]


#13526

Fromawnl <awnl@gmx-topmail.de>
Date2013-02-18 10:59 +0000
Message-ID<kft1ij$cov$1@smc.vnet.net>
In reply to#13515
Am 17.02.2013 10:08, schrieb David Bailey:
> On 16/02/2013 06:07, Ramiro Barrantes wrote:
>> Hello,
>>
>> I would like to get a random line from a file, I know this can be done
>> with Mathematica but I am playing with using sed to see if it goes
>> faster, say I want to get line 1000
>>
>> In mathematica it would be:
>>
>> <<"! sed -n p1000 filename.txt"
>>
>> However, I am trying to put the filename as a variable, say
>>
>> filename="hugefile.txt"
>>
>> cmd="! sed -n p1000 "<>filename
>> <<cmd
>>
>> does not work.
>>
>> How can I do this?
>>
>> Lastly, I am getting a randomline using mathematica doing:
>>
>> getRandomLine[file_, n_] :=
>>     Block[{i = RandomInteger[{1, n}], str = OpenRead[file], res},
>>      Skip[str, "String", i];
>>      res = Read[str, Expression];
>>      Close[str];
>>      res[[2]]
>>      ]
>>
>> However, it is very slow so I was going to try with sed.Any suggestions?
>>
>> Thanks in advance,
>> Ramiro
>>
>>
> I would stick with Mathematica to do this job! How big is the file
> (number of lines and number of bytes)? If it will fit inside Mathematica
> comfortable, I'd see how it works to read it all in as a list of strings
> and pick the one you want:
>
> xx=ReadList["C:\\some file",String];//Timing
>
> Then you have an array of strings, and you can select what you want
> directly.
>
> Remember, the basic problem with reading at an arbitrary position in a
> text file, is that if the line lengths are not the same, any algorithm
> has to read every line before the one you want!

if he just wants to get an arbitrary line that's not true: just choosing 
a position in the file at random and searching e.g. the previous and 
next linebreak would also result in picking a random line. Of course the 
probability of choosing longer lines would be larger than that for 
shorter lines, but it isn't clear from the question whether that would 
be a problem for what the OP tries to do...

> If you create this file,
> you should consider packing the lines to make them all the same length -
> then you could access what you want very efficiently (but with a little
> more coding!)

... and slightly (?) higher memory requirements...

hth,

albert

[toc] | [prev] | [next] | [standalone]


#13550

FromDavid Bailey <dave@removedbailey.co.uk>
Date2013-02-19 06:09 +0000
Message-ID<kfv4v0$3ft$1@smc.vnet.net>
In reply to#13526
On 18/02/2013 10:59, awnl wrote:
> Am 17.02.2013 10:08, schrieb David Bailey:
>> On 16/02/2013 06:07, Ramiro Barrantes wrote:
>>> Hello,
>>>
>>> I would like to get a random line from a file, I know this can be done
>>> with Mathematica but I am playing with using sed to see if it goes
>>> faster, say I want to get line 1000
>>>
>>> In mathematica it would be:
>>>
>>> <<"! sed -n p1000 filename.txt"
>>>
>>> However, I am trying to put the filename as a variable, say
>>>
>>> filename="hugefile.txt"
>>>
>>> cmd="! sed -n p1000 "<>filename
>>> <<cmd
>>>
>>> does not work.
>>>
>>> How can I do this?
>>>
>>> Lastly, I am getting a randomline using mathematica doing:
>>>
>>> getRandomLine[file_, n_] :=
>>>      Block[{i = RandomInteger[{1, n}], str = OpenRead[file], res},
>>>       Skip[str, "String", i];
>>>       res = Read[str, Expression];
>>>       Close[str];
>>>       res[[2]]
>>>       ]
>>>
>>> However, it is very slow so I was going to try with sed.Any suggestions?
>>>
>>> Thanks in advance,
>>> Ramiro
>>>
>>>
>> I would stick with Mathematica to do this job! How big is the file
>> (number of lines and number of bytes)? If it will fit inside Mathematica
>> comfortable, I'd see how it works to read it all in as a list of strings
>> and pick the one you want:
>>
>> xx=ReadList["C:\\some file",String];//Timing
>>
>> Then you have an array of strings, and you can select what you want
>> directly.
>>
>> Remember, the basic problem with reading at an arbitrary position in a
>> text file, is that if the line lengths are not the same, any algorithm
>> has to read every line before the one you want!
>
> if he just wants to get an arbitrary line that's not true: just choosing
> a position in the file at random and searching e.g. the previous and
> next linebreak would also result in picking a random line. Of course the
> probability of choosing longer lines would be larger than that for
> shorter lines, but it isn't clear from the question whether that would
> be a problem for what the OP tries to do...
>
>> If you create this file,
>> you should consider packing the lines to make them all the same length -
>> then you could access what you want very efficiently (but with a little
>> more coding!)
>
> ... and slightly (?) higher memory requirements...
>
> hth,
>
> albert
>
Well yes - but I assume that by 'arbitrary' he means a specific record 
somewhere in the file!

David Bailey
http://www.dbaileyconsultancy.co.uk

[toc] | [prev] | [next] | [standalone]


#13552

FromRamiro <ramiro.barrantes@gmail.com>
Date2013-02-19 06:09 +0000
Message-ID<kfv4vg$3g3$1@smc.vnet.net>
In reply to#13515
Thank you so much for the reply.  My files are 50MB each, I don't think ReadList would work for my purposes, it would be too slow.  I am actually doing an MCMC simulation, doing (hopefully if I have time) millions of iterations and in each one I need to read a random line from one of many files, thus requiring this reading to happen as quickly as possible. Any suggestions? Each line is pretty much the same length.

Thanks,
Ramiro

On Sunday, February 17, 2013 4:08:27 AM UTC-5, David Bailey wrote:
> On 16/02/2013 06:07, Ramiro Barrantes wrote:
>
> > Hello,
>
> >
>
> > I would like to get a random line from a file, I know this can be done
>
> > with Mathematica but I am playing with using sed to see if it goes
>
> > faster, say I want to get line 1000
>
> >
>
> > In mathematica it would be:
>
> >
>
> > <<"! sed -n p1000 filename.txt"
>
> >
>
> > However, I am trying to put the filename as a variable, say
>
> >
>
> > filename="hugefile.txt"
>
> >
>
> > cmd="! sed -n p1000 "<>filename
>
> > <<cmd
>
> >
>
> > does not work.
>
> >
>
> > How can I do this?
>
> >
>
> > Lastly, I am getting a randomline using mathematica doing:
>
> >
>
> > getRandomLine[file_, n_] :=
>
> >    Block[{i = RandomInteger[{1, n}], str = OpenRead[file], res},
>
> >     Skip[str, "String", i];
>
> >     res = Read[str, Expression];
>
> >     Close[str];
>
> >     res[[2]]
>
> >     ]
>
> >
>
> > However, it is very slow so I was going to try with sed.Any suggestions?
>
> >
>
> > Thanks in advance,
>
> > Ramiro
>
> >
>
> >
>
> I would stick with Mathematica to do this job! How big is the file
>
> (number of lines and number of bytes)? If it will fit inside Mathematica
>
> comfortable, I'd see how it works to read it all in as a list of strings
>
> and pick the one you want:
>
>
>
> xx=ReadList["C:\\some file",String];//Timing
>
>
>
> Then you have an array of strings, and you can select what you want
>
> directly.
>
>
>
> Remember, the basic problem with reading at an arbitrary position in a
>
> text file, is that if the line lengths are not the same, any algorithm
>
> has to read every line before the one you want! If you create this file,
>
> you should consider packing the lines to make them all the same length -
>
> then you could access what you want very efficiently (but with a little
>
> more coding!)
>
>
>
> David Bailey
>
> http://www.dbaileyconsultancy.co.uk

[toc] | [prev] | [next] | [standalone]


#13560

From"Kevin J. McCann" <kjm@KevinMcCann.com>
Date2013-02-19 23:51 +0000
Message-ID<kg136k$85n$1@smc.vnet.net>
In reply to#13552
If you plan to do this millions of times, then your only hope is to load 
the file(s) into memory, e.g. with ReadList. If you do a disk access for 
each line, you will be waiting for quite a while. Memory is cheap.

Kevin

On 2/19/2013 1:09 AM, Ramiro wrote:
> Thank you so much for the reply.  My files are 50MB each, I don't think ReadList would work for my purposes, it would be too slow.  I am actually doing an MCMC simulation, doing (hopefully if I have time) millions of iterations and in each one I need to read a random line from one of many files, thus requiring this reading to happen as quickly as possible. Any suggestions? Each line is pretty much the same length.
>
> Thanks,
> Ramiro
>
> On Sunday, February 17, 2013 4:08:27 AM UTC-5, David Bailey wrote:
>> On 16/02/2013 06:07, Ramiro Barrantes wrote:
>>
>>> Hello,
>>
>>>
>>
>>> I would like to get a random line from a file, I know this can be done
>>
>>> with Mathematica but I am playing with using sed to see if it goes
>>
>>> faster, say I want to get line 1000
>>
>>>
>>
>>> In mathematica it would be:
>>
>>>
>>
>>> <<"! sed -n p1000 filename.txt"
>>
>>>
>>
>>> However, I am trying to put the filename as a variable, say
>>
>>>
>>
>>> filename="hugefile.txt"
>>
>>>
>>
>>> cmd="! sed -n p1000 "<>filename
>>
>>> <<cmd
>>
>>>
>>
>>> does not work.
>>
>>>
>>
>>> How can I do this?
>>
>>>
>>
>>> Lastly, I am getting a randomline using mathematica doing:
>>
>>>
>>
>>> getRandomLine[file_, n_] :=
>>
>>>     Block[{i = RandomInteger[{1, n}], str = OpenRead[file], res},
>>
>>>      Skip[str, "String", i];
>>
>>>      res = Read[str, Expression];
>>
>>>      Close[str];
>>
>>>      res[[2]]
>>
>>>      ]
>>
>>>
>>
>>> However, it is very slow so I was going to try with sed.Any suggestions?
>>
>>>
>>
>>> Thanks in advance,
>>
>>> Ramiro
>>
>>>
>>
>>>
>>
>> I would stick with Mathematica to do this job! How big is the file
>>
>> (number of lines and number of bytes)? If it will fit inside Mathematica
>>
>> comfortable, I'd see how it works to read it all in as a list of strings
>>
>> and pick the one you want:
>>
>>
>>
>> xx=ReadList["C:\\some file",String];//Timing
>>
>>
>>
>> Then you have an array of strings, and you can select what you want
>>
>> directly.
>>
>>
>>
>> Remember, the basic problem with reading at an arbitrary position in a
>>
>> text file, is that if the line lengths are not the same, any algorithm
>>
>> has to read every line before the one you want! If you create this file,
>>
>> you should consider packing the lines to make them all the same length -
>>
>> then you could access what you want very efficiently (but with a little
>>
>> more coding!)
>>
>>
>>
>> David Bailey
>>
>> http://www.dbaileyconsultancy.co.uk
>
>

[toc] | [prev] | [next] | [standalone]


#13563

Fromawnl <awnl@gmx-topmail.de>
Date2013-02-19 23:52 +0000
Message-ID<kg137f$86c$1@smc.vnet.net>
In reply to#13552
Hi,

> Thank you so much for the reply.  My files are 50MB each, I don't
> think ReadList would work for my purposes, it would be too slow.  I
> am actually doing an MCMC simulation, doing (hopefully if I have
> time) millions of iterations and in each one I need to read a random
> line from one of many files, thus requiring this reading to happen as
> quickly as possible. Any suggestions? Each line is pretty much the
> same length.

For that specific use case I see two possible ways to proceed:

1) pick a random position in the file, search line break before that 
position and linebreak after that position, read that line. The 
drawbacks of that approach is that the probability to pick a line is 
proportional to its length and that you have to search line start and 
end for each line you read. If all lines have the same length there is 
no problem, if they are different it depends on whether your MC 
simulation will suffer from a non-uniform distribution or not...

2) build an index of line-starts. Instead of reading the full content 
you could just scan through the file once, searching the positions where 
new lines start and build a list of file position with length equal to 
the number of lines. Then you'd have to choose one of these position 
(e.g. with RandomChoice), seek that position and read one line. I guess 
that this is probably the best alternative. To speed up the building of 
the index you might want to read the file in chunks of several lines 
instead of line by line. You could have a look at e.g. this 
mathematica.stackexchange answer 
http://mathematica.stackexchange.com/a/15216/169
for an example on how to do that.

Whichever way you choose, you will need the low level functions for file 
access: search for OpenRead, StreamPosition, SetStreamPosition, Read, 
ReadList and Close in the documentation for details about these...

hth,

albert



[toc] | [prev] | [next] | [standalone]


#13564

FromDavid Bailey <dave@removedbailey.co.uk>
Date2013-02-19 23:52 +0000
Message-ID<kg1380$86i$1@smc.vnet.net>
In reply to#13552
On 19/02/2013 06:09, Ramiro wrote:
> Thank you so much for the reply.  My files are 50MB each, I don't think ReadList would work for my purposes, it would be too slow.  I am actually doing an MCMC simulation, doing (hopefully if I have time) millions of iterations and in each one I need to read a random line from one of many files, thus requiring this reading to happen as quickly as possible. Any suggestions? Each line is pretty much the same length.
>
> Thanks,
> Ramiro
>

OK - let's establish two points:

1)      Are the records in the files of a fixed length?

2)      When you say you want an 'arbitrary line' I am assuming that you 
calculate a number N, and when want the N'th line of the file. If you 
really don't care which line you choose, use Ramiro's method (above).

If your files are not guaranteed to have equal length records, there is 
obviously a problem, as I explained before, because you have to read all 
N-1 lines to establish which is the N'th. One option therefore, might be 
to pre-process your files to make fixed length records by padding with 
blanks.

Once you have fixed record length files, you can open them with 
BinaryFormat->True and use SetStreamPosition to set the stream to the 
position in bytes where your record starts, and read the relevant number 
of bytes. Unless you are using extended characters, you could convert 
these to characters with FromCharacterCode.

This should be VERY fast, because the cost of each access is not 
proportional to the size of the file (once all the files have been 
preprocessed).

If the records are variable length but contain some identification such 
as a line number, another option would be to pull out a line as Ramiro 
suggested, but then use a binary chop procedure to zero in on the line 
of interest.

Hint: You may want to look at the processed file with a hex editor, to 
make sure the record length is as you expect - remember Windows uses 2 
characters per end of line!

David Bailey
http://www.dbaileyconsultancy.co.uk

[toc] | [prev] | [standalone]


Back to top | Article view | comp.soft-sys.math.mathematica


csiph-web