Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #90996 > unrolled thread

Find if a file existing within 1000s of folder/sub-folder - each file has a unique presence

Back to article view | Back to comp.lang.python

Contents

  Find if a file existing within 1000s of folder/sub-folder - each file has a unique presence chaotic.sid@gmail.com - 2015-05-20 22:34 -0700
    Re: Find if a file existing within 1000s of folder/sub-folder - each file has a unique presence Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-05-21 18:07 +1000
      Re: Find if a file existing within 1000s of folder/sub-folder - each file has a unique presence Peter Otten <__peter__@web.de> - 2015-05-21 10:54 +0200
      Re: Find if a file existing within 1000s of folder/sub-folder - each file has a unique presence Tim Golden <mail@timgolden.me.uk> - 2015-05-21 10:06 +0100
      Re: Find if a file existing within 1000s of folder/sub-folder - each file has a unique presence Grant Edwards <invalid@invalid.invalid> - 2015-05-21 14:14 +0000
        Re: Find if a file existing within 1000s of folder/sub-folder - each file has a unique presence Chris Angelico <rosuav@gmail.com> - 2015-05-22 00:27 +1000
          Re: Find if a file existing within 1000s of folder/sub-folder - each file has a unique presence Grant Edwards <invalid@invalid.invalid> - 2015-05-21 16:31 +0000
            Re: Find if a file existing within 1000s of folder/sub-folder - each file has a unique presence MRAB <python@mrabarnett.plus.com> - 2015-05-21 17:54 +0100
        Re: Find if a file existing within 1000s of folder/sub-folder - each file has a unique presence Tim Golden <mail@timgolden.me.uk> - 2015-05-21 15:32 +0100

#90996 — Find if a file existing within 1000s of folder/sub-folder - each file has a unique presence

Hi All,

I have a list of scripts which are present in various folders and this may be present in one of their sub-folder too.
There are 1000s of folders and hence parsing through each folder/sub-folder is not an option.

So I was trying to dir /s /b using python.
Now since the file's path name is computed using other part of the code, I am feeding in a variable here and somehow it does not seem to work.. :(

Intent is to run following command from python 
dir /s /b "c:/abc/def/ghjmain\features\XYZ\*<filename>"

My present code.
import subprocess

for row in range(1,max_row):
	tempID_cell = "C" + str(row);
	tempID = ws[tempID_cell].value;
	print (tempID);
	Command = "c:\\abc\\def\\ghj\\main\\features\\XYZ\\*" + str(tempID) + ".cli";
	print (Command);
	IsPresent = subprocess.check_output("dir /s /b Command", shell=True);

Any help would really be appreciated.

Regards,
Siddharth

[toc] | [next] | [standalone]

#91001

On Thursday 21 May 2015 15:34, chaotic.sid@gmail.com wrote:

> So I was trying to dir /s /b using python.
> Now since the file's path name is computed using other part of the code, I
> am feeding in a variable here and somehow it does not seem to work.. :(
> 
> Intent is to run following command from python
> dir /s /b "c:/abc/def/ghjmain\features\XYZ\*<filename>"

I don't use Windows and have no idea what the /s and /b flags do, but 
something like this should work:

import glob
print(glob.glob("c:/abc/def/ghjmain/features/XYZ/*<filename>"))


Don't use backslashes \ as they have special meaning to Python. Use forward 
slashes and let Python convert them as needed.

Also, this only looks in the C:.../XYZ directory. I think that Python 3.4 or 
better will accept a ** wildcard instead of * to look in subdirectories too. 
Or you can use a simple directory walker:

# Untested.
import os, os.path
start = "c:/abc/def/ghjmain/features/XYZ/"
for dirpath, dirnames, filenames in os.walk(start):
    for name in filenames:
        if name.endswith("filename"):
            print(os.path.join(dirpath, name))


If you can do the file listing from Python without calling out to an 
external process, it will likely be faster.



> My present code.
> import subprocess
> 
> for row in range(1,max_row):
>     tempID_cell = "C" + str(row);
>     tempID = ws[tempID_cell].value;
>     print (tempID);
>     Command = "c:\\abc\\def\\ghj\\main\\features\\XYZ\\*" + str(tempID) +
>     ".cli"; print (Command);
>     IsPresent = subprocess.check_output("dir /s /b Command", shell=True);
> 
> Any help would really be appreciated.


Start by defining "somehow it does not seem to work" (your words, above). 
What happens? Does your computer crash? It downloads files off the Internet? 
Raises an exception? Prints the wrong information?

If it raises an exception, you should COPY AND PASTE the full traceback. If 
it prints an error message from dir, you should do the same.

What happens when you take the command you generate and run it in the shell?

I think what you want is:

path = "c:/abc/def/ghj/main/features/XYZ/*" + str(tempID) + ".cli"
command = "dir /s /b " + path
output = subprocess.check_output(command, shell=True)

Actually, I think you don't need the shell argument. Try this instead:

path = "c:/abc/def/ghj/main/features/XYZ/*" + str(tempID) + ".cli"
output = subprocess.check_output(['dir', '/s', '/b', path])

That is likely to be safer and will help avoid shell injection attacks if 
the path comes from an untrusted source. Any subprocess experts want to 
confirm this?



P.S. Python is not C or Javascript. There is no need to end each line with a 
semicolon. That just makes you seem like a n00b.


-- 
Steve

[toc] | [prev] | [next] | [standalone]

#91003

Steven D'Aprano wrote:

> Also, this only looks in the C:.../XYZ directory. I think that Python 3.4
> or better will accept a ** wildcard instead of * to look in subdirectories
> too. 

Unfortunately that's Python 3.5. You are ahead of time.

> Actually, I think you don't need the shell argument. Try this instead:
> 
> path = "c:/abc/def/ghj/main/features/XYZ/*" + str(tempID) + ".cli"
> output = subprocess.check_output(['dir', '/s', '/b', path])

"dir" used to be an internal command; back in the day you would have to 
invoke it with (untested)

check_output(["cmd", "/k", 'dir', '/s', '/b', path])

I believe on Windows path may contain wildcards -- or does subprocess escape 
these somehow? 

> That is likely to be safer and will help avoid shell injection attacks if 
> the path comes from an untrusted source. Any subprocess experts want to 
> confirm this?
 
I probably should have kept quiet to let a combined subprocess/windows 
expert chime in -- but now it's too late...

[toc] | [prev] | [next] | [standalone]

#91004

On 21/05/2015 09:07, Steven D'Aprano wrote:
> On Thursday 21 May 2015 15:34, chaotic.sid@gmail.com wrote:
> 
>> So I was trying to dir /s /b using python.
>> Now since the file's path name is computed using other part of the code, I
>> am feeding in a variable here and somehow it does not seem to work.. :(
>>
>> Intent is to run following command from python
>> dir /s /b "c:/abc/def/ghjmain\features\XYZ\*<filename>"
> 
> I don't use Windows and have no idea what the /s and /b flags do, but 
> something like this should work:

Just for the uninitiated: /b shows only filenames ("bare") and /s shows
subdirectories.

[ ... snip os.walk example ...]

> path = "c:/abc/def/ghj/main/features/XYZ/*" + str(tempID) + ".cli"
> command = "dir /s /b " + path
> output = subprocess.check_output(command, shell=True)
> 
> Actually, I think you don't need the shell argument. Try this instead:

This is one of the few cases on Windows where you actually *do* need the
shell=True. shell=True invokes "cmd.exe" which is needed for the
command, such as dir and copy, which aren't standalone
executables but subcommands of cmd.exe.

I agree with Steven that os.walk, or something derived from it, is
definitely the way to go here, unless the OP has tried it and found it
to be too slow. Clearly, some kind of cacheing would help since the idea
seems to be to find, one at a time, a filename in any one of a large
hierarchy of directories.

Python 3.5 has the brand-new os.scandir which does a more efficient job
than os.listdir, using the underlying OS facilities on each platform and
cacheing where possible. But that's really leading edge, although Ben
Hoyt (the author) maintains an up-to-date version on github:

https://github.com/benhoyt/scandir


Just for the exercise, here's code which builds a dictionary mapping
filename to location(s) found:

<code>
#!python3
import os, sys

START_FROM = "c:/temp"

files = {}
for dirpath, dirnames, filenames in os.walk(START_FROM):
	for filename in filenames:
		files.setdefault(filename, []).append(dirpath)

filename_to_find = "temp.txt" ## input("Filename: ")
found_in = files.get(filename_to_find)
if found_in:
	print("Found in: ", ", ".join(found_in))
else:
	print("Not found")

</code>


TJG

[toc] | [prev] | [next] | [standalone]

#91009

On 2015-05-21, Steven D'Aprano <steve+comp.lang.python@pearwood.info> wrote:

> import glob
> print(glob.glob("c:/abc/def/ghjmain/features/XYZ/*<filename>"))
>
> Don't use backslashes \ as they have special meaning to Python. Use forward 
> slashes and let Python convert them as needed.

Interesting. I've never heard about this.

When will Python convert them?

-- 
Grant Edwards               grant.b.edwards        Yow! I was born in a
                                  at               Hostess Cupcake factory
                              gmail.com            before the sexual
                                                   revolution!

[toc] | [prev] | [next] | [standalone]

#91010

On Fri, May 22, 2015 at 12:14 AM, Grant Edwards <invalid@invalid.invalid> wrote:
> On 2015-05-21, Steven D'Aprano <steve+comp.lang.python@pearwood.info> wrote:
>
>> import glob
>> print(glob.glob("c:/abc/def/ghjmain/features/XYZ/*<filename>"))
>>
>> Don't use backslashes \ as they have special meaning to Python. Use forward
>> slashes and let Python convert them as needed.
>
> Interesting. I've never heard about this.
>
> When will Python convert them?

Actually, it won't ever bother to convert them. The Windows file
system APIs are quite happy to work with forward slashes; it's only
command-line tools (which conventionally use forward slashes to
introduce options), and not all of them, which require backslashes.

You may want to consider explicitly converting them in your own code,
prior to showing a path to a human; but even back in the 1990s, it
wasn't uncommon for cross-platform programs to mix and match - for
instance, if you unzip something into C:\Foo\Bar, you'd get output
like "Inflating C:\Foo\Bar/usr/lib/whatever". I'd have no objections
to a program using forward slashes all the way.

ChrisA

[toc] | [prev] | [next] | [standalone]

#91014

On 2015-05-21, Chris Angelico <rosuav@gmail.com> wrote:
> On Fri, May 22, 2015 at 12:14 AM, Grant Edwards <invalid@invalid.invalid> wrote:
>> On 2015-05-21, Steven D'Aprano <steve+comp.lang.python@pearwood.info> wrote:
>>
>>> import glob
>>> print(glob.glob("c:/abc/def/ghjmain/features/XYZ/*<filename>"))
>>>
>>> Don't use backslashes \ as they have special meaning to Python. Use forward
>>> slashes and let Python convert them as needed.
>>
>> Interesting. I've never heard about this.
>>
>> When will Python convert them?
>
> Actually, it won't ever bother to convert them.

OK, so this isn't some new feature I hadn't heard about due to my
spending most of my time with 2.7. :)

> The Windows file system APIs are quite happy to work with forward
> slashes;

Yep, I knew that -- I have always use forward slashes on Windows (and
DOS before that) when dealing with the file system.

> it's only command-line tools (which conventionally use forward
> slashes to introduce options), and not all of them, which require
> backslashes.

Yup, I was wondering if that was where Python (or its stdlib) would
convert them (which would have surprised me).  Back in the day, you
could change the 'option switch' character from '/' to whatever you
wanted (and as an old Unix guy, I always set it to '-').  Then you
could even use forward slashes on the command line (mostly). But, I
don't think Windows has support that for yonks.

-- 
Grant Edwards               grant.b.edwards        Yow! Please come home with
                                  at               me ... I have Tylenol!!
                              gmail.com            

[toc] | [prev] | [next] | [standalone]

#91017

On 2015-05-21 17:31, Grant Edwards wrote:
> On 2015-05-21, Chris Angelico <rosuav@gmail.com> wrote:
>> On Fri, May 22, 2015 at 12:14 AM, Grant Edwards <invalid@invalid.invalid> wrote:
>>> On 2015-05-21, Steven D'Aprano <steve+comp.lang.python@pearwood.info> wrote:
>>>
>>>> import glob
>>>> print(glob.glob("c:/abc/def/ghjmain/features/XYZ/*<filename>"))
>>>>
>>>> Don't use backslashes \ as they have special meaning to Python. Use forward
>>>> slashes and let Python convert them as needed.
>>>
>>> Interesting. I've never heard about this.
>>>
>>> When will Python convert them?
>>
>> Actually, it won't ever bother to convert them.
>
> OK, so this isn't some new feature I hadn't heard about due to my
> spending most of my time with 2.7. :)
>
>> The Windows file system APIs are quite happy to work with forward
>> slashes;
>
> Yep, I knew that -- I have always use forward slashes on Windows (and
> DOS before that) when dealing with the file system.
>
>> it's only command-line tools (which conventionally use forward
>> slashes to introduce options), and not all of them, which require
>> backslashes.
>
Dialog boxes, however, insist on backslashses.

> Yup, I was wondering if that was where Python (or its stdlib) would
> convert them (which would have surprised me).  Back in the day, you
> could change the 'option switch' character from '/' to whatever you
> wanted (and as an old Unix guy, I always set it to '-').  Then you
> could even use forward slashes on the command line (mostly). But, I
> don't think Windows has support that for yonks.
>

[toc] | [prev] | [next] | [standalone]

#91011

On 21/05/2015 15:14, Grant Edwards wrote:
> On 2015-05-21, Steven D'Aprano <steve+comp.lang.python@pearwood.info> wrote:
> 
>> import glob
>> print(glob.glob("c:/abc/def/ghjmain/features/XYZ/*<filename>"))
>>
>> Don't use backslashes \ as they have special meaning to Python. Use forward 
>> slashes and let Python convert them as needed.
> 
> Interesting. I've never heard about this.
> 
> When will Python convert them?
> 

It doesn't: Python passes them on unchanged and Windows accepts them in
all but a few cases.

(Although if any stdlib function were to call, eg, os.path.abspath on a
path before passing it along to Windows then one effect is that the path
is "normalised", ending up with backslashes. So perhaps that does happen
in modules like shutil. Not sure).


TJG

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web