Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.php > #12710 > unrolled thread

I Need to search over 100 largeish text documents efficiently. What's the best approach?

Back to article view | Back to comp.lang.php

Contents

  I Need to search over 100 largeish  text documents efficiently. What's the best approach? rob.bradford2805@gmail.com - 2014-01-26 05:34 -0800
    Re: I Need to search over 100 largeish  text documents efficiently. What's the best approach? Richard Damon <Richard@Damon-Family.org> - 2014-01-26 09:09 -0500
      Re: I Need to search over 100 largeish  text documents efficiently. What's the best approach? Michael Vilain <vilain@NOspamcop.net> - 2014-01-26 13:34 -0800
    Re: I Need to search over 100 largeish  text documents efficiently. What's the best approach? Jerry Stuckle <jstucklex@attglobal.net> - 2014-01-26 10:29 -0500
      Re: I Need to search over 100 largeish  text documents efficiently. What's the best approach? Denis McMahon <denismfmcmahon@gmail.com> - 2014-01-26 20:14 +0000
    Re: I Need to search over 100 largeish  text documents efficiently. What's the best approach? The Natural Philosopher <tnp@invalid.invalid> - 2014-01-26 19:56 +0000
    Re: I Need to search over 100 largeish  text documents efficiently. What's the best approach? Ben Bacarisse <ben.usenet@bsb.me.uk> - 2014-01-26 20:55 +0000
    Re: I Need to search over 100 largeish  text documents efficiently. What's the best approach? Denis McMahon <denismfmcmahon@gmail.com> - 2014-01-27 01:43 +0000
      Re: I Need to search over 100 largeish  text documents efficiently.   What's the best approach? Arno Welzel <usenet@arnowelzel.de> - 2014-01-27 10:58 +0100
        Re: I Need to search over 100 largeish  text documents efficiently.   What's the best approach? Denis McMahon <denismfmcmahon@gmail.com> - 2014-01-27 12:23 +0000
        Re: I Need to search over 100 largeish  text documents efficiently. What's the best approach? Ben Bacarisse <ben.usenet@bsb.me.uk> - 2014-01-27 17:05 +0000
    Re: I Need to search over 100 largeish  text documents efficiently. What's the best approach? rob.bradford2805@gmail.com - 2014-01-27 13:14 -0800

#12710 — I Need to search over 100 largeish text documents efficiently. What's the best approach?

As part of my hosting providers re-platforming cycle my site has moved server, on the new server and all new servers php exec() and equivalents are blocked, this has taken out my fast document search that used exec() to call grep then awk.  I now need to do the grep part as effectively as possible in PHP as I can no longer access the shell from the scripts. The awk part is easily sorted.

What is the best/fastest approach to scan 100+ largish text files for word strings, I really don't wish to index each file into a database as the documents change quite frequently. my grep-awk scan was around one second to begin rendering the results page, I know I can't match that but I can't afford too much of a delay.

Any ideas appreciated whilst I look for a new hosting provider, I feel that any hosting set up that makes such a change without notification really has no respect for it's clients.

Rob

[toc] | [next] | [standalone]

#12711

On 1/26/14, 8:34 AM, rob.bradford2805@gmail.com wrote:
> As part of my hosting providers re-platforming cycle my site has moved server, on the new server and all new servers php exec() and equivalents are blocked, this has taken out my fast document search that used exec() to call grep then awk.  I now need to do the grep part as effectively as possible in PHP as I can no longer access the shell from the scripts. The awk part is easily sorted.
> 
> What is the best/fastest approach to scan 100+ largish text files for word strings, I really don't wish to index each file into a database as the documents change quite frequently. my grep-awk scan was around one second to begin rendering the results page, I know I can't match that but I can't afford too much of a delay.
> 
> Any ideas appreciated whilst I look for a new hosting provider, I feel that any hosting set up that makes such a change without notification really has no respect for it's clients.
> 
> Rob
> 

If you can't call grep from the command line via exec, the best solution
may be to write a version of grep in your program. Read the files (or
chunks of them in sequence) and use the PHP string search functions on
the data block. If reading chunks, make sure to do any needed overlap
between chunks so you don't miss matches across chunk breaks.

[toc] | [prev] | [next] | [standalone]

#12716

In article <SG8Fu.46700$vG7.15374@en-nntp-03.dc1.easynews.com>,
 Richard Damon <Richard@Damon-Family.org> wrote:

> On 1/26/14, 8:34 AM, rob.bradford2805@gmail.com wrote:
> > As part of my hosting providers re-platforming cycle my site has moved 
> > server, on the new server and all new servers php exec() and equivalents 
> > are blocked, this has taken out my fast document search that used exec() to 
> > call grep then awk.  I now need to do the grep part as effectively as 
> > possible in PHP as I can no longer access the shell from the scripts. The 
> > awk part is easily sorted.
> > 
> > What is the best/fastest approach to scan 100+ largish text files for word 
> > strings, I really don't wish to index each file into a database as the 
> > documents change quite frequently. my grep-awk scan was around one second 
> > to begin rendering the results page, I know I can't match that but I can't 
> > afford too much of a delay.
> > 
> > Any ideas appreciated whilst I look for a new hosting provider, I feel that 
> > any hosting set up that makes such a change without notification really has 
> > no respect for it's clients.
> > 
> > Rob
> > 
> 
> If you can't call grep from the command line via exec, the best solution
> may be to write a version of grep in your program. Read the files (or
> chunks of them in sequence) and use the PHP string search functions on
> the data block. If reading chunks, make sure to do any needed overlap
> between chunks so you don't miss matches across chunk breaks.

If you're greping multiple large files from an exec, grep will produced 
the results on each file as it's processed.  You can easily replicate 
this behavior from within php.

Loop through an array containing the filenames you want to grep.
In each file, open it and read it into memory as an array.
use preg_grep or preg_match_all to scan the entire array for results.
do whatever you want with the resultant array of matching results.
process the next file.  That seems fairly straightforward. 

If these files are HUGE (e.g. GB), you may have to do your own I/O with 
fopen/fread, convert the string buffer into an array with split, grep 
it, and get more data.  The problem there is you may pull in a partial 
line.

-- 
DeeDee, don't press that button!  DeeDee!  NO!  Dee...
[I filter all Goggle Groups posts, so any reply may be automatically ignored]

[toc] | [prev] | [next] | [standalone]

#12712

On 1/26/2014 8:34 AM, rob.bradford2805@gmail.com wrote:
> As part of my hosting providers re-platforming cycle my site has moved server, on the new server and all new servers php exec() and equivalents are blocked, this has taken out my fast document search that used exec() to call grep then awk.  I now need to do the grep part as effectively as possible in PHP as I can no longer access the shell from the scripts. The awk part is easily sorted.
>
> What is the best/fastest approach to scan 100+ largish text files for word strings, I really don't wish to index each file into a database as the documents change quite frequently. my grep-awk scan was around one second to begin rendering the results page, I know I can't match that but I can't afford too much of a delay.
>
> Any ideas appreciated whilst I look for a new hosting provider, I feel that any hosting set up that makes such a change without notification really has no respect for it's clients.
>
> Rob
>

Whether the files change frequently or not, your best bet is going to be 
putting the documents in a database.  You won't be able to do anything 
nearly as fast in PHP as the database does.  And it isn't that hard to 
insert the documents into the database when they are uploaded.

And kudos to your hosting provider for closing a huge security exposure.

-- 
==================
Remove the "x" from my email address
Jerry Stuckle
jstucklex@attglobal.net
==================

[toc] | [prev] | [next] | [standalone]

#12714

On Sun, 26 Jan 2014 10:29:58 -0500, Jerry Stuckle wrote:

> On 1/26/2014 8:34 AM, rob.bradford2805@gmail.com wrote:

>> Any ideas appreciated whilst I look for a new hosting provider, I feel
>> that any hosting set up that makes such a change without notification
>> really has no respect for it's clients.

> And kudos to your hosting provider for closing a huge security exposure.

+1 to both these comments.

Your hosting provider should have told you about this change as soon as 
they became aware it would happen (and if they weren't aware they'd be 
forcing this change on their customers, then they need to be shot because 
that would mean that the people managing the server switch had no idea of 
the effect of the differences in configurations).

However, you'd have been even more pissed at your hosting provider if 
your website had suddenly started serving up porn or russian mafia 
viruses or viagra-clone adverts to all visitors because the now closed 
security hole had been exploited.

-- 
Denis McMahon, denismfmcmahon@gmail.com

[toc] | [prev] | [next] | [standalone]

#12713

On 26/01/14 13:34, rob.bradford2805@gmail.com wrote:
> As part of my hosting providers re-platforming cycle my site has moved server, on the new server and all new servers php exec() and equivalents are blocked, this has taken out my fast document search that used exec() to call grep then awk.  I now need to do the grep part as effectively as possible in PHP as I can no longer access the shell from the scripts. The awk part is easily sorted.
>
> What is the best/fastest approach to scan 100+ largish text files for word strings, I really don't wish to index each file into a database as the documents change quite frequently. my grep-awk scan was around one second to begin rendering the results page, I know I can't match that but I can't afford too much of a delay.
>
> Any ideas appreciated whilst I look for a new hosting provider, I feel that any hosting set up that makes such a change without notification really has no respect for it's clients.
>
> Rob
>
change service providers and get your own virtual machine.


-- 
Ineptocracy

(in-ep-toc’-ra-cy) – a system of government where the least capable to 
lead are elected by the least capable of producing, and where the 
members of society least likely to sustain themselves or succeed, are 
rewarded with goods and services paid for by the confiscated wealth of a 
diminishing number of producers.

[toc] | [prev] | [next] | [standalone]

#12715

rob.bradford2805@gmail.com writes:

> As part of my hosting providers re-platforming cycle my site has moved
> server, on the new server and all new servers php exec() and
> equivalents are blocked, this has taken out my fast document search
> that used exec() to call grep then awk.  I now need to do the grep
> part as effectively as possible in PHP as I can no longer access the
> shell from the scripts. The awk part is easily sorted.
>
> What is the best/fastest approach to scan 100+ largish text files for
> word strings, I really don't wish to index each file into a database
> as the documents change quite frequently. my grep-awk scan was around
> one second to begin rendering the results page, I know I can't match
> that but I can't afford too much of a delay.

I second Richard Damon's suggestion.  If the awk bit is easily sorted,
and all you are missing is grep, it must be a matter of minutes to make
something like the functionality you had before from a handful of lines
of PHP[1].  Of course, it won't be an external command, so the way it
integrates with the rest of the code might make this not quite the
trivial task that it first appears to be.

[1] fgets, preg_match and glob (if you need it).

-- 
Ben.

[toc] | [prev] | [next] | [standalone]

#12717

On Sun, 26 Jan 2014 05:34:21 -0800, rob.bradford2805 wrote:

> What is the best/fastest approach to scan 100+ largish text files for
> word strings

A quick googling finds:

http://sourceforge.net/projects/php-grep/
http://net-wrench.com/download-tools/php-grep.php

Claims to be able to search 1000 files in under 10 secs

-- 
Denis McMahon, denismfmcmahon@gmail.com

[toc] | [prev] | [next] | [standalone]

#12718 — Re: I Need to search over 100 largeish text documents efficiently. What's the best approach?

Am 27.01.2014 02:43, schrieb Denis McMahon:

> On Sun, 26 Jan 2014 05:34:21 -0800, rob.bradford2805 wrote:
> 
>> What is the best/fastest approach to scan 100+ largish text files for
>> word strings
> 
> A quick googling finds:
> 
> http://sourceforge.net/projects/php-grep/
> http://net-wrench.com/download-tools/php-grep.php
> 
> Claims to be able to search 1000 files in under 10 secs

Under ideal conditions - maybe. But if each file is more than 1 MB, it
is barely possible to even read this amount of data in just 10 seconds
(assuming around 80 MB/s and 1000 MB of data to be searched).

Even using a simple word index (word plus the name of the file(s) and
the position(s) where the word is located) would be the better solution.


-- 
Arno Welzel
http://arnowelzel.de
http://de-rec-fahrrad.de

[toc] | [prev] | [next] | [standalone]

#12719 — Re: I Need to search over 100 largeish text documents efficiently. What's the best approach?

On Mon, 27 Jan 2014 10:58:42 +0100, Arno Welzel wrote:

> Am 27.01.2014 02:43, schrieb Denis McMahon:
> 
>> On Sun, 26 Jan 2014 05:34:21 -0800, rob.bradford2805 wrote:
>> 
>>> What is the best/fastest approach to scan 100+ largish text files for
>>> word strings
>> 
>> A quick googling finds:
>> 
>> http://sourceforge.net/projects/php-grep/
>> http://net-wrench.com/download-tools/php-grep.php
>> 
>> Claims to be able to search 1000 files in under 10 secs
> 
> Under ideal conditions - maybe. But if each file is more than 1 MB, it
> is barely possible to even read this amount of data in just 10 seconds
> (assuming around 80 MB/s and 1000 MB of data to be searched).
> 
> Even using a simple word index (word plus the name of the file(s) and
> the position(s) where the word is located) would be the better solution.

Indeed, the fastest solution would be to index each file when it changes, 
and keep the indexes in a db.

Perhaps there are common words you wouldn't index, in english these might 
include:

a the in on an this that then ....

Then if you have a search phrase, remove the common words, look for the 
uncommon words in close proximity to each other

It might help to know more about the grep too, is this using complex 
regexp, or is it a simple string search done externally using grep.

-- 
Denis McMahon, denismfmcmahon@gmail.com

[toc] | [prev] | [next] | [standalone]

#12720

Arno Welzel <usenet@arnowelzel.de> writes:

> Am 27.01.2014 02:43, schrieb Denis McMahon:
>
>> On Sun, 26 Jan 2014 05:34:21 -0800, rob.bradford2805 wrote:
>> 
>>> What is the best/fastest approach to scan 100+ largish text files for
>>> word strings
>> 
>> A quick googling finds:
>> 
>> http://sourceforge.net/projects/php-grep/
>> http://net-wrench.com/download-tools/php-grep.php
>> 
>> Claims to be able to search 1000 files in under 10 secs
>
> Under ideal conditions - maybe. But if each file is more than 1 MB, it
> is barely possible to even read this amount of data in just 10 seconds
> (assuming around 80 MB/s and 1000 MB of data to be searched).

There are so many variable here; we don't know what "largeish" means and
we don't know what sort of grep is being done (to count lines, display
lines, or just to find matching files?).

Anyway, I tried a couple of there out.  With the most naive PHP grep
imaginable, I can do the equivalent of grep -l (to find matching files)
in reasonable time.  With 987M of data in 201 5M files, it takes about
0.5 second in PHP compared to about 0.15 seconds for grep.  I recon if
that is what is being done, then it's well worth it as a temporary
solution.

Counting matching lines is much slower: about 17s for PHP vs 1s for
grep.  Although the server is likely to be faster than my laptop, the
ratios might be useful data.

-- 
Ben.

[toc] | [prev] | [next] | [standalone]

#12721

On Sunday, 26 January 2014 13:34:21 UTC, rob.brad...@gmail.com  wrote:
> As part of my hosting providers re-platforming cycle my site has moved server, on the new server and all new servers php exec() and equivalents are blocked, this has taken out my fast document search that used exec() to call grep then awk.  I now need to do the grep part as effectively as possible in PHP as I can no longer access the shell from the scripts. The awk part is easily sorted.
> 
> 
> 
> What is the best/fastest approach to scan 100+ largish text files for word strings, I really don't wish to index each file into a database as the documents change quite frequently. my grep-awk scan was around one second to begin rendering the results page, I know I can't match that but I can't afford too much of a delay.
> 
> 
> 
> Any ideas appreciated whilst I look for a new hosting provider, I feel that any hosting set up that makes such a change without notification really has no respect for it's clients.
> 
> 
> 
> Rob

Thanks for all the suggestions, and my gripe with the hosting provider is not that they closed the security risk, just the lack of notice that it was going to happen, then leaving me to wonder why stuff had stopped working.

Rob

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.php

csiph-web