Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.php > #12719

Re: I Need to search over 100 largeish text documents efficiently. What's the best approach?

Show all headers | View raw

On Mon, 27 Jan 2014 10:58:42 +0100, Arno Welzel wrote:

> Am 27.01.2014 02:43, schrieb Denis McMahon:
> 
>> On Sun, 26 Jan 2014 05:34:21 -0800, rob.bradford2805 wrote:
>> 
>>> What is the best/fastest approach to scan 100+ largish text files for
>>> word strings
>> 
>> A quick googling finds:
>> 
>> http://sourceforge.net/projects/php-grep/
>> http://net-wrench.com/download-tools/php-grep.php
>> 
>> Claims to be able to search 1000 files in under 10 secs
> 
> Under ideal conditions - maybe. But if each file is more than 1 MB, it
> is barely possible to even read this amount of data in just 10 seconds
> (assuming around 80 MB/s and 1000 MB of data to be searched).
> 
> Even using a simple word index (word plus the name of the file(s) and
> the position(s) where the word is located) would be the better solution.

Indeed, the fastest solution would be to index each file when it changes, 
and keep the indexes in a db.

Perhaps there are common words you wouldn't index, in english these might 
include:

a the in on an this that then ....

Then if you have a search phrase, remove the common words, look for the 
uncommon words in close proximity to each other

It might help to know more about the grep too, is this using complex 
regexp, or is it a simple string search done externally using grep.

-- 
Denis McMahon, denismfmcmahon@gmail.com

Back to comp.lang.php | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

I Need to search over 100 largeish  text documents efficiently. What's the best approach? rob.bradford2805@gmail.com - 2014-01-26 05:34 -0800
  Re: I Need to search over 100 largeish  text documents efficiently. What's the best approach? Richard Damon <Richard@Damon-Family.org> - 2014-01-26 09:09 -0500
    Re: I Need to search over 100 largeish  text documents efficiently. What's the best approach? Michael Vilain <vilain@NOspamcop.net> - 2014-01-26 13:34 -0800
  Re: I Need to search over 100 largeish  text documents efficiently. What's the best approach? Jerry Stuckle <jstucklex@attglobal.net> - 2014-01-26 10:29 -0500
    Re: I Need to search over 100 largeish  text documents efficiently. What's the best approach? Denis McMahon <denismfmcmahon@gmail.com> - 2014-01-26 20:14 +0000
  Re: I Need to search over 100 largeish  text documents efficiently. What's the best approach? The Natural Philosopher <tnp@invalid.invalid> - 2014-01-26 19:56 +0000
  Re: I Need to search over 100 largeish  text documents efficiently. What's the best approach? Ben Bacarisse <ben.usenet@bsb.me.uk> - 2014-01-26 20:55 +0000
  Re: I Need to search over 100 largeish  text documents efficiently. What's the best approach? Denis McMahon <denismfmcmahon@gmail.com> - 2014-01-27 01:43 +0000
    Re: I Need to search over 100 largeish  text documents efficiently.   What's the best approach? Arno Welzel <usenet@arnowelzel.de> - 2014-01-27 10:58 +0100
      Re: I Need to search over 100 largeish  text documents efficiently.   What's the best approach? Denis McMahon <denismfmcmahon@gmail.com> - 2014-01-27 12:23 +0000
      Re: I Need to search over 100 largeish  text documents efficiently. What's the best approach? Ben Bacarisse <ben.usenet@bsb.me.uk> - 2014-01-27 17:05 +0000
  Re: I Need to search over 100 largeish  text documents efficiently. What's the best approach? rob.bradford2805@gmail.com - 2014-01-27 13:14 -0800

csiph-web