Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #41185 > unrolled thread

IOError:[Errno 27] File too large

Back to article view | Back to comp.lang.python

Contents

  IOError:[Errno 27] File too large ch.valderanis@gmail.com - 2013-03-13 09:53 -0700
    Re: IOError:[Errno 27] File too large Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-03-13 17:32 +0000
      Re: IOError:[Errno 27] File too large ch.valderanis@gmail.com - 2013-03-13 15:35 -0700
        Re: IOError:[Errno 27] File too large Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-03-14 02:46 +0000
          Re: IOError:[Errno 27] File too large ch.valderanis@gmail.com - 2013-03-14 13:12 -0700
            Re: IOError:[Errno 27] File too large Dave Angel <davea@davea.name> - 2013-03-14 21:32 -0400
    Re: IOError:[Errno 27] File too large Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2013-03-13 18:27 -0400

#41185 — IOError:[Errno 27] File too large

Hi,

Relatively newcomer here.
The following code fails with the above error:
python version used 2.6.2 under linux

filestring='somestring'
for files in glob.glob('*'):
	f2=open(files.replace('.xml','.sub'),'w')
	f2.write(filestring+files)
	f2.close()


The glob commands returns around 10k files.
I want to create 10k new files. the filename should have a different extension (sub instead of xml)and the content of the new file should be some filestring with the addition of the original filename.

I am pretty sure that the files is less than 1kB. Is there another reason for the operation to fail?

Any other comments,remarks welcomed. 

Thanks,
Makis

[toc] | [next] | [standalone]

#41186

On Wed, 13 Mar 2013 09:53:17 -0700, ch.valderanis wrote:

> Hi,
> 
> Relatively newcomer here.
> The following code fails with the above error: python version used 2.6.2
> under linux

Which part of the code fails? Please copy and paste the entire traceback, 
starting with the line "Traceback (most recent call last)" and ending 
with the complete error message.

Which file does it fail on?


> filestring='somestring'
> for files in glob.glob('*'):
> 	f2=open(files.replace('.xml','.sub'),'w') 
> 	f2.write(filestring+files)
> 	f2.close()

Some other comments:

You use the name "files" to represent a single filename, rather than 
multiple files. That is misleading, a poor choice of name.

"f2" is also a poor name.

You try to change the file extension using string.replace method. This is 
risky, and will fail if some file has ".xml" in the filename apart from 
the file extension. Worse, if you have a file *without* ".xml", your code 
will over-write that file. Better to use the os.path.splitext function 
for this.

Is filestring really a constant string like in the snippet above? Is 
there any change that it could be an enormous string?


Re-writing your code snippet:


import os
filestring = 'somestring'
for filename in glob.glob('*.xml'):
    newname = os.path.splitext(filename)[0] + '.sub'
    newfile = open(newname, 'w')
    newfile.write(filestring + filename)
    newfile.close()



> I am pretty sure that the files is less than 1kB.

"Pretty sure"? You need to be 100% certain. We can't tell you whether 
this is the case or not.



> Is there another reason for the operation to fail?

We don't know which operation has failed. That's why you need to show the 
complete traceback.




-- 
Steven

[toc] | [prev] | [next] | [standalone]

#41197

Dear Steven,

Thank you very much both for your answer and of course your comments. They are taken into account.
I found out that when I touch FILENAME.sub in the command line, I get the same error. So I guess it isn't a problem with the language but rather deeper. I will solve this first before I come back.
Just for completeness:

> Which part of the code fails? Please copy and paste the entire traceback, 
> starting with the line "Traceback (most recent call last)" and ending 
> with the complete error message.
The code is corrected according to your advice:
Here it the error:

Traceback (most recent call last):
  File "createsubmitfiles.py", line 12, in <module>
    newfile=open(newname,'w')
IOError: [Errno 27] File too large: 'FILENAME.sub;'
 
I have changed the actual filename reported by the traceback.
FILENAME.xml does exist before the code is run
FILENAME.sub is not created
FILENAME has a length of 160 characters
I do have space to write all the files in the directory

> Is filestring really a constant string like in the snippet above? Is 
> there any change that it could be an enormous string?
Yes. It is a constant string
 
 
> > I am pretty sure that the files is less than 1kB.
> "Pretty sure"? You need to be 100% certain. We can't tell you whether 
> this is the case or not.
When I do an ls -l before I run the code and after the code has failed, all I see are files with size <1kB.

Makis

[toc] | [prev] | [next] | [standalone]

#41208

On Wed, 13 Mar 2013 15:35:19 -0700, ch.valderanis wrote:

> Traceback (most recent call last):
>   File "createsubmitfiles.py", line 12, in <module>
>     newfile=open(newname,'w')
> IOError: [Errno 27] File too large: 'FILENAME.sub;'
>  
> I have changed the actual filename reported by the traceback.
> FILENAME.xml does exist before the code is run FILENAME.sub is not
> created
> FILENAME has a length of 160 characters I do have space to write all the
> files in the directory

You are not trying to create a file called "FILENAME.sub". You are trying 
to create a file called "FILENAME.sub;", notice the semi-colon.

Taking a wild guess, I think that you are using a Samba share on a Linux 
server. A file "FILENAME.xml;" was accidentally creating on this share 
from the Linux filesystem layer, since Linux will allow you to use 
semicolons in file names. But samba enforces the same restrictions as 
Windows, so when you access the file system through sambda, it gives an 
error when you try to create a new file "FILENAME.sub;".

This is a wild guess, and could be completely wrong. Please come back and 
tell us the solution when you have solved it.



Good luck!



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#41240

> Taking a wild guess, I think that you are using a Samba share on a Linux 
> 
> server. A file "FILENAME.xml;" was accidentally creating on this share 
> 
> from the Linux filesystem layer, since Linux will allow you to use 
> 
> semicolons in file names. But samba enforces the same restrictions as 
> 
> Windows, so when you access the file system through sambda, it gives an 
> 
> error when you try to create a new file "FILENAME.sub;".
> 
> This is a wild guess, and could be completely wrong. Please come back and 
> 
> tell us the solution when you have solved it.

So, let me report the solution.
All the files were in an afs filesystem. afs has a limit for the total length of  the filenames per directory. for more details take a look at http://www.inf.ed.ac.uk/systems/AFS/topten.html#Tip13
In my case 10k files with ~160characters per filename were just hitting the limit. The error message is basically misleading.

Makis

[toc] | [prev] | [next] | [standalone]

#41254

On 03/14/2013 04:12 PM, ch.valderanis@gmail.com wrote:
>
>> Taking a wild guess, I think that you are using a Samba share on a Linux
>>
>> server. A file "FILENAME.xml;" was accidentally creating on this share
>>
>> from the Linux filesystem layer, since Linux will allow you to use
>>
>> semicolons in file names. But samba enforces the same restrictions as
>>
>> Windows, so when you access the file system through sambda, it gives an
>>
>> error when you try to create a new file "FILENAME.sub;".
>>
>> This is a wild guess, and could be completely wrong. Please come back and
>>
>> tell us the solution when you have solved it.
>
> So, let me report the solution.
> All the files were in an afs filesystem. afs has a limit for the total length of  the filenames per directory. for more details take a look at http://www.inf.ed.ac.uk/systems/AFS/topten.html#Tip13
> In my case 10k files with ~160characters per filename were just hitting the limit. The error message is basically misleading.
>
> Makis
>

Aha!  So the file that was too large was the directory file.

-- 
DaveA

[toc] | [prev] | [next] | [standalone]

#41196

On Wed, 13 Mar 2013 09:53:17 -0700 (PDT), ch.valderanis@gmail.com
declaimed the following in gmane.comp.python.general:

> Hi,
> 
> Relatively newcomer here.
> The following code fails with the above error:
> python version used 2.6.2 under linux
> 
> filestring='somestring'
> for files in glob.glob('*'):
> 	f2=open(files.replace('.xml','.sub'),'w')

	"Danger Will Robinson, Danger"

	What do you expect to have happen for files that did NOT have ".xml"
somewhere in the name. If you only want to do this for *.xml you should
probably ask for that in the glob() call.

	Also, the standard library has functions for splitting file names
into prefix and extension...

> 	f2.write(filestring+files)
> 	f2.close()

	<snip> 

> I am pretty sure that the files is less than 1kB. Is there another reason for the operation to fail?
> 
> Any other comments,remarks welcomed. 
>
	Provide the full traceback, so we can confirm what line produced the
error.
-- 
	Wulfraed                 Dennis Lee Bieber         AF6VN
        wlfraed@ix.netcom.com    HTTP://wlfraed.home.netcom.com/

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web