Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.java.programmer > #10412 > unrolled thread

java.io.File

Back to article view | Back to comp.lang.java.programmer

Contents

  java.io.File Mark <i@dontgetlotsofspamanymore.net> - 2011-12-02 09:35 +0000
    Re: java.io.File Lew <lewbloch@gmail.com> - 2011-12-02 07:25 -0800
      Re: java.io.File Mark <i@dontgetlotsofspamanymore.net> - 2011-12-02 15:55 +0000
        Re: java.io.File Lew <lewbloch@gmail.com> - 2011-12-02 08:17 -0800
          Re: java.io.File Mark <i@dontgetlotsofspamanymore.net> - 2011-12-02 16:31 +0000
            Re: java.io.File markspace <-@.> - 2011-12-02 08:53 -0800
            Re: java.io.File Lew <lewbloch@gmail.com> - 2011-12-02 14:02 -0800
              Re: java.io.File Arne Vajhøj <arne@vajhoej.dk> - 2011-12-02 17:12 -0500
                Re: java.io.File Lew <lewbloch@gmail.com> - 2011-12-02 14:16 -0800
                  Re: java.io.File Arne Vajhøj <arne@vajhoej.dk> - 2011-12-02 18:42 -0500
                    Re: java.io.File Lars Enderin <lars.enderin@telia.com> - 2011-12-03 01:06 +0100
                  Re: java.io.File Jukka Lahtinen <jtfjdehf@hotmail.com.invalid> - 2011-12-07 15:51 +0200
              Re: java.io.File Mark <i@dontgetlotsofspamanymore.net> - 2011-12-05 15:14 +0000
                Re: java.io.File markspace <-@.> - 2011-12-05 07:49 -0800
                  Re: java.io.File Mark <i@dontgetlotsofspamanymore.net> - 2011-12-05 16:28 +0000
                Re: java.io.File Lew <lewbloch@gmail.com> - 2011-12-05 15:25 -0800
                  Re: java.io.File Mark <i@dontgetlotsofspamanymore.net> - 2011-12-06 10:29 +0000
            Re: java.io.File Arne Vajhøj <arne@vajhoej.dk> - 2011-12-02 17:08 -0500
        Re: java.io.File Roedy Green <see_website@mindprod.com.invalid> - 2011-12-03 00:57 -0800
      Re: java.io.File Mayeul <mayeul.marguet@free.fr> - 2011-12-02 17:33 +0100
        Re: java.io.File Lew <lewbloch@gmail.com> - 2011-12-02 14:07 -0800
          Re: java.io.File Mayeul <mayeul.marguet@free.fr> - 2011-12-06 15:52 +0100
            Re: java.io.File Lew <lewbloch@gmail.com> - 2011-12-06 09:23 -0800

Page 1 of 2  [1] 2  Next page →

#10412 — java.io.File

Can a java.io.File object use a OS file descriptor?  I am trying to
find the source of a fd leak in a application.

[toc] | [next] | [standalone]

#10424

Mark wrote:
> Can a java.io.File object use a OS file descriptor?  I am trying to
> find the source of a fd leak in a[n] application.

At some point, depending on the operations performed by the 'File' instance, there may be a file descriptor involved, and then the 'File' instance certainly does use it, at least indirectly via JVM system calls that proxy to OS system calls.

From a Java perspective you should look for unclosed I/O streams/channels and packratted 'File' instances rather than file descriptors.

Of course I'm shooting in the dark since you've said absolutely nothing about your application, much less provided an http://sscce.org/.

-- 
Lew

[toc] | [prev] | [next] | [standalone]

#10426

On Fri, 2 Dec 2011 07:25:02 -0800 (PST), Lew <lewbloch@gmail.com>
wrote:

>Mark wrote:
>> Can a java.io.File object use a OS file descriptor?  I am trying to
>> find the source of a fd leak in a[n] application.
>
>At some point, depending on the operations performed by the 'File' instance, 
>there may be a file descriptor involved, and then the 'File' instance certainly
>does use it, at least indirectly via JVM system calls that proxy to OS system
>calls.
>
>From a Java perspective you should look for unclosed I/O streams/channels and 
>packratted 'File' instances rather than file descriptors.

I've done a code inspection and the streams are all explicitly closed.
There are a number of File objects used and I notice that File does
not have a close() method so we have to rely on GC.

>Of course I'm shooting in the dark since you've said absolutely nothing about
>your application, much less provided an http://sscce.org/.

I know.  It would be difficult to produce an example since I cannot
reproduce the problem myself - it happens in production only and I
don't have any access there.

I get the exception:
java.io.IOException: Cannot run program "/path/to/script":
java.io.IOException: error=24, Too many open files

The script is called via ProcessBuilder object and I don't see any
obvious problems there.

[toc] | [prev] | [next] | [standalone]

#10428

Mark wrote:
> Lew wrote:
>> Mark wrote:
>>> Can a java.io.File object use a OS file descriptor?  I am trying to
>>> find the source of a fd leak in a[n] application.
>>
>> At some point, depending on the operations performed by the 'File' instance, 
>> there may be a file descriptor involved, and then the 'File' instance certainly
>> does use it, at least indirectly via JVM system calls that proxy to OS system
>> calls.
>>
>> From a Java perspective you should look for unclosed I/O streams/channels and 
>> packratted 'File' instances rather than file descriptors.
> 
> I've done a code inspection and the streams are all explicitly closed.
> There are a number of File objects used and I notice that File does
> not have a close() method so we have to rely on GC.

If there were a 'close()' method, as there is with streams, it would have nothing to do with GC.  'close()' is for resources (such as file handles).  GC is for heap memory.  I only suggested checking for packratted 'File' instances as a foolish guess.  Now that I think about it, it is highly unlikely that unclaimed instances would have anything to do with your issue.

>> Of course I'm shooting in the dark since you've said absolutely nothing about
>> your application, much less provided an http://sscce.org/.
> 
> I know.  It would be difficult to produce an example since I cannot
> reproduce the problem myself - it happens in production only and I
> don't have any access there.
> 
> I get the exception:
> java.io.IOException: Cannot run program "/path/to/script":
> java.io.IOException: error=24, Too many open files

Do you have any reason to believe it's your program that holds those files?

> The script is called via ProcessBuilder object and I don't see any
> obvious problems there.

Or subtle ones either, I guess.

An irreproducible problem involving disparate runtime environments (inside and outside the JVM), executing black-box code on an undisclosed platform, yielding an error message possibly originated by an entity outside the JVM with no data forthcoming - is that the issue with which you request assistance?

Note:  the preceding paragraph implies a path forward, in that a problem statement illuminates prospective solutions.  In case you missed it, the path forward is for you to gain visibility into that plethora of unknowns, or sadly report to your client, as we do to you, that with nothing to go on you have nothing to offer.

Visibility involves things like logging.  You do use logging, don't you?

As an extra benefit, the experience of using logs to diagnose a production issue will provide an object lesson into how useless most programmers are at designing logging, and give you insight into proper logging practices.

-- 
Lew

[toc] | [prev] | [next] | [standalone]

#10430

On Fri, 2 Dec 2011 08:17:15 -0800 (PST), Lew <lewbloch@gmail.com>
wrote:

(BTW your posts appear to have no line wrapping)

>Mark wrote:
>> Lew wrote:
>>> Mark wrote:
>>>> Can a java.io.File object use a OS file descriptor?  I am trying to
>>>> find the source of a fd leak in a[n] application.
>>>
>>> At some point, depending on the operations performed by the 'File' instance, 
>>> there may be a file descriptor involved, and then the 'File' instance certainly
>>> does use it, at least indirectly via JVM system calls that proxy to OS system
>>> calls.
>>>
>>> From a Java perspective you should look for unclosed I/O streams/channels and 
>>> packratted 'File' instances rather than file descriptors.
>> 
>> I've done a code inspection and the streams are all explicitly closed.
>> There are a number of File objects used and I notice that File does
>> not have a close() method so we have to rely on GC.
>
>If there were a 'close()' method, as there is with streams, it would have 
>nothing to do with GC.  'close()' is for resources (such as file handles).
>GC is for heap memory.  I only suggested checking for packratted 'File'
>instances as a foolish guess.  Now that I think about it, it is highly
>unlikely that unclaimed instances would have anything to do with your issue.

AFAIK many classes have a close() method to allow any underlying OS
resources to be explicitly freed without needing to wait for the
dispose() method to do this.  If the File method does uses file
descriptors then we may assume that these could be left open until the
object is destroyed during GC.

>>> Of course I'm shooting in the dark since you've said absolutely nothing about
>>> your application, much less provided an http://sscce.org/.
>> 
>> I know.  It would be difficult to produce an example since I cannot
>> reproduce the problem myself - it happens in production only and I
>> don't have any access there.
>> 
>> I get the exception:
>> java.io.IOException: Cannot run program "/path/to/script":
>> java.io.IOException: error=24, Too many open files
>
>Do you have any reason to believe it's your program that holds those files?

The reporter of the problem believes so.

>> The script is called via ProcessBuilder object and I don't see any
>> obvious problems there.
>
>Or subtle ones either, I guess.

Correct.

>An irreproducible problem involving disparate runtime environments (inside
>and outside the JVM), executing black-box code on an undisclosed platform
>yielding an error message possibly originated by an entity outside the JVM
>with no data forthcoming - is that the issue with which you request assistance?

The error message did orginate within the JVM.  I know the problem is
a tricky one which is why I specificly worded it to relate to the
java.io.File class.

>Note:  the preceding paragraph implies a path forward, in that a problem statement
>illuminates prospective solutions.  In case you missed it, the path forward is
>for you to gain visibility into that plethora of unknowns, or sadly report
>to your client, as we do to you, that with nothing to go on you have nothing to offer.

Unfortunately I work in the real world.  I often get reported problems
with inadequate information.  Usually I can work out what is going on
even though it's like doing a jigsaw puzzle with most of the pieces
missing.  I don't like telling clients that I have nothing to offer in
case they tell me the same ;-)

>Visibility involves things like logging.  You do use logging, don't you?

The author of the code did use logging.  It's not sufficiently
detailed to be of much use though.

[toc] | [prev] | [next] | [standalone]

#10431

On 12/2/2011 8:31 AM, Mark wrote:

> On Fri, 2 Dec 2011 08:17:15 -0800 (PST), Lew<lewbloch@gmail.com>
> wrote:
>> If there were a 'close()' method, as there is with streams, it would have
>> nothing to do with GC.  'close()' is for resources (such as file handles).
>> GC is for heap memory.  I only suggested checking for packratted 'File'
>> instances as a foolish guess.  Now that I think about it, it is highly
>> unlikely that unclaimed instances would have anything to do with your issue.


> AFAIK many classes have a close() method to allow any underlying OS
> resources to be explicitly freed without needing to wait for the
> dispose() method to do this.  If the File method does uses file
> descriptors then we may assume that these could be left open until the
> object is destroyed during GC.


What Mayeul said.

File is just a pathname, so of course it has no "close" method.  You 
should be looking for FileInputStream, InputStream, OutputStream, 
RandomAccessFile, and all their subclasses (which there are quite a 
few).  Just looking for File won't cut it, as most of those classes take 
other objects besides File to construct a stream.  For example, 
FileInputStream can also be constructed from a path in a String.

Maybe you knew this and just misspoke, but it seemed like a point of 
confusion.

[toc] | [prev] | [next] | [standalone]

#10437

Mark wrote:
> Lew wrote:
> 
> (BTW your posts appear to have no line wrapping)

I cannot control how my posts appear in your newsreader.

Your lines appear not to wrap in my newsreader also.

> >Mark wrote:
>>> Lew wrote:
>>>> Mark wrote:
>>>>> Can a java.io.File object use a OS file descriptor?  I am trying to
>>>>> find the source of a fd leak in a[n] application.
>>>>
>>>> At some point, depending on the operations performed by the 'File' instance, 
>>>> there may be a file descriptor involved, and then the 'File' instance certainly
>>>> does use it, at least indirectly via JVM system calls that proxy to OS system
>>>> calls.
>>>>
>>>> From a Java perspective you should look for unclosed I/O streams/channels and 
>>>> packratted 'File' instances rather than file descriptors.
>>> 
>>> I've done a code inspection and the streams are all explicitly closed.
>>> There are a number of File objects used and I notice that File does
>>> not have a close() method so we have to rely on GC.
>>
>> If there were a 'close()' method, as there is with streams, it would have 
>> nothing to do with GC.  'close()' is for resources (such as file handles).
>> GC is for heap memory.  I only suggested checking for packratted 'File'
>> instances as a foolish guess.  Now that I think about it, it is highly
>> unlikely that unclaimed instances would have anything to do with your issue.
> 
> AFAIK many classes have a close() method to allow any underlying OS
> resources to be explicitly freed without needing to wait for the
> dispose() method to do this.  If the File method does uses file

What 'dispose()' method?  'File' doesn't have one.  'FileOutputStream' doesn't have one.  'FileInputStream' doesn't have one.  None of their ancestor classes up through 'Object' has one.

> descriptors then we may assume that these could be left open until the
> object is destroyed during GC.

No, we cannot assume any such thing, nor can we even assume any given 'File' instance, or any other class instance, will be collected ever.

'FileInputStream', but not all 'InputStream' types, and 'FileOutputStream', but not all 'OutputStream' types, do have a 'finalize()' method that might, eventually, release file resources assuming the instances go through two GC cycles, but there's no assurance that that will happen.

>>>> Of course I'm shooting in the dark since you've said absolutely nothing about
>>>> your application, much less provided an http://sscce.org/.
>>> 
>>> I know.  It would be difficult to produce an example since I cannot
>>> reproduce the problem myself - it happens in production only and I
>>> don't have any access there.
>>> 
>>> I get the exception:
>>> java.io.IOException: Cannot run program "/path/to/script":
>>> java.io.IOException: error=24, Too many open files
>>
>> Do you have any reason to believe it's your program that holds those files?
> 
> The reporter of the problem believes so.

Some people believe in Creationism.  It doesn't make their belief valid.

You need evidence.

>>> The script is called via ProcessBuilder object and I don't see any
>>> obvious problems there.
>>
>> Or subtle ones either, I guess.
> 
> Correct.
> 
>> An irreproducible problem involving disparate runtime environments (inside
>> and outside the JVM), executing black-box code on an undisclosed platform
>> yielding an error message possibly originated by an entity outside the JVM
>> with no data forthcoming - is that the issue with which you request assistance?
> 
> The error message did orginate within the JVM.  I know the problem is
> a tricky one which is why I specificly worded it to relate to the
> java.io.File class.

The message originated in the JVM; it doesn't mean that the problem did.

>> Note:  the preceding paragraph implies a path forward, in that a problem statement
>> illuminates prospective solutions.  In case you missed it, the path forward is
>> for you to gain visibility into that plethora of unknowns, or sadly report
>> to your client, as we do to you, that with nothing to go on you have nothing to offer.
> 
> Unfortunately I work in the real world.  I often get reported problems

My advice is for the real world.  Duh.

> with inadequate information.  Usually I can work out what is going on

There's a difference between inadequate information and zero information.  You have given us zero information.

> even though it's like doing a jigsaw puzzle with most of the pieces
> missing.  I don't like telling clients that I have nothing to offer in
> case they tell me the same ;-)

That's why I gave you advice on how to gather more information.  You have to know /something/ in order to fix the problem, don't you?

>> Visibility involves things like logging.  You do use logging, don't you?
> 
> The author of the code did use logging.  It's not sufficiently
> detailed to be of much use though.

Thus illustrating my point that programmers write terrible logs.

I hear your admirable commitment to do some good, but honestly you need a lot more information than you have shared here to even begin.  I also hear your rejection of my advice to gather such information.  Good luck with that.

-- 
Lew

[toc] | [prev] | [next] | [standalone]

#10441

On 12/2/2011 5:02 PM, Lew wrote:
> Mark wrote:
>> Lew wrote:
>>
>> (BTW your posts appear to have no line wrapping)
>
> I cannot control how my posts appear in your newsreader.

No.

But by choice of your newsreader and the way you enter text
you can control whether what is send to the NNTP server has
CRLF's embedded at approx. 80 characters or not.

I suspect that it has not.

Arne

[toc] | [prev] | [next] | [standalone]

#10442

On Friday, December 2, 2011 2:12:54 PM UTC-8, Arne Vajhøj wrote:
> On 12/2/2011 5:02 PM, Lew wrote:
> > Mark wrote:
> >> Lew wrote:
> >>
> >> (BTW your posts appear to have no line wrapping)
> >
> > I cannot control how my posts appear in your newsreader.
> 
> No.
> 
> But by choice of your newsreader and the way you enter text
> you can control whether what is send to the NNTP server has
> CRLF's embedded at approx. 80 characters or not.
> 
> I suspect that it has not.

I notice that the guy who complained had the exact same thing happen in his posts, so I don't take the complaint at all seriously.

I also notice that my posts wrap in my newsreader.

-- 
Lew

[toc] | [prev] | [next] | [standalone]

#10445

On 12/2/2011 5:16 PM, Lew wrote:
> On Friday, December 2, 2011 2:12:54 PM UTC-8, Arne Vajhøj wrote:
>> On 12/2/2011 5:02 PM, Lew wrote:
>>> Mark wrote:
>>>> Lew wrote:
>>>>
>>>> (BTW your posts appear to have no line wrapping)
>>>
>>> I cannot control how my posts appear in your newsreader.
>>
>> No.
>>
>> But by choice of your newsreader and the way you enter text
>> you can control whether what is send to the NNTP server has
>> CRLF's embedded at approx. 80 characters or not.
>>
>> I suspect that it has not.
>
> I notice that the guy who complained had the exact same thing happen in his posts, so I don't take the complaint at all seriously.
>
> I also notice that my posts wrap in my newsreader.

Your posts does not wrap in TB.

Everybody elses posts do and yours used to do so.

Arne

[toc] | [prev] | [next] | [standalone]

#10446

2011-12-03 00:42, Arne Vajhøj skrev:
> On 12/2/2011 5:16 PM, Lew wrote:
>> On Friday, December 2, 2011 2:12:54 PM UTC-8, Arne Vajhøj wrote:
>>> On 12/2/2011 5:02 PM, Lew wrote:
>>>> Mark wrote:
>>>>> Lew wrote:
>>>>>
>>>>> (BTW your posts appear to have no line wrapping)
>>>>
>>>> I cannot control how my posts appear in your newsreader.
>>>
>>> No.
>>>
>>> But by choice of your newsreader and the way you enter text
>>> you can control whether what is send to the NNTP server has
>>> CRLF's embedded at approx. 80 characters or not.
>>>
>>> I suspect that it has not.
>>
>> I notice that the guy who complained had the exact same thing happen
>> in his posts, so I don't take the complaint at all seriously.
>>
>> I also notice that my posts wrap in my newsreader.
> 
> Your posts does not wrap in TB.
> 
> Everybody elses posts do and yours used to do so.

He's using GG. GG uses Quoted Printable encoding if it finds a reason to
do so, which seems to include the 8-bit character ø in your name, and
also using "=" to break too long lines for transmission. Like this:
------
> CRLF's embedded at approx. 80 characters or not.
>=20
> I suspect that it has not.

I notice that the guy who complained had the exact same thing happen in his=
 posts, so I don't take the complaint at all seriously.
------
But I have seen earlier GG posts from Lew with long lines and no QP
encoding.

[toc] | [prev] | [next] | [standalone]

#10590

Lew <lewbloch@gmail.com> writes:

>> > Mark wrote:

>> >> (BTW your posts appear to have no line wrapping)

> I notice that the guy who complained had the exact same thing happen in his posts, so I don't take the complaint at all seriously.

There is no  format=flowed definition in your Content-Type heading.
That's why your posgings don't wrap without the newlines.

-- 
Jukka Lahtinen

[toc] | [prev] | [next] | [standalone]

#10528

On Fri, 2 Dec 2011 14:02:05 -0800 (PST), Lew <lewbloch@gmail.com>
wrote:

>Mark wrote:
>> Lew wrote:
>> 
>> >Mark wrote:
>>>> Lew wrote:
>>>>> Mark wrote:
>>>>>> Can a java.io.File object use a OS file descriptor?  I am trying to
>>>>>> find the source of a fd leak in a[n] application.
>>>>>
>>>>> At some point, depending on the operations performed by the 'File' instance, 
>>>>> there may be a file descriptor involved, and then the 'File' instance certainly
>>>>> does use it, at least indirectly via JVM system calls that proxy to OS system
>>>>> calls.
>>>>>
>>>>> From a Java perspective you should look for unclosed I/O streams/channels and 
>>>>> packratted 'File' instances rather than file descriptors.
>>>> 
>>>> I've done a code inspection and the streams are all explicitly closed.
>>>> There are a number of File objects used and I notice that File does
>>>> not have a close() method so we have to rely on GC.
>>>
>>> If there were a 'close()' method, as there is with streams, it would have 
>>> nothing to do with GC.  'close()' is for resources (such as file handles).
>>> GC is for heap memory.  I only suggested checking for packratted 'File'
>>> instances as a foolish guess.  Now that I think about it, it is highly
>>> unlikely that unclaimed instances would have anything to do with your issue.
>> 
>> AFAIK many classes have a close() method to allow any underlying OS
>> resources to be explicitly freed without needing to wait for the
>> dispose() method to do this.  If the File method does uses file
>
>What 'dispose()' method?  'File' doesn't have one.  'FileOutputStream' doesn't have
>one.  'FileInputStream' doesn't have one.  None of their ancestor classes up through
>'Object' has one.
>
>> descriptors then we may assume that these could be left open until the
>> object is destroyed during GC.
>
>No, we cannot assume any such thing, nor can we even assume any given 'File' instance
>, or any other class instance, will be collected ever.

Unless the Java class has been badly written we can assume that
when/if the object is collected then any OS resources that it used
would also be freed.  Would you agree with this?

We also see that File does not have a close(), dispose() or similar
method.  Therefore, if the object does use any OS resources, there is
no way for the programmer to force them to be released.

>'FileInputStream', but not all 'InputStream' types, and 'FileOutputStream', but
>not all 'OutputStream' types, do have a 'finalize()' method that might, eventually,
>release file resources assuming the instances go through two GC cycles, but there's
>no assurance that that will happen.

Which, presumably, they have close() methods defined.

>>>>> Of course I'm shooting in the dark since you've said absolutely nothing about
>>>>> your application, much less provided an http://sscce.org/.
>>>> 
>>>> I know.  It would be difficult to produce an example since I cannot
>>>> reproduce the problem myself - it happens in production only and I
>>>> don't have any access there.
>>>> 
>>>> I get the exception:
>>>> java.io.IOException: Cannot run program "/path/to/script":
>>>> java.io.IOException: error=24, Too many open files
>>>
>>> Do you have any reason to believe it's your program that holds those files?
>> 
>> The reporter of the problem believes so.
>
>Some people believe in Creationism.  It doesn't make their belief valid.

Oh Boy.

>You need evidence.
>
>>>> The script is called via ProcessBuilder object and I don't see any
>>>> obvious problems there.
>>>
>>> Or subtle ones either, I guess.
>> 
>> Correct.
>> 
>>> An irreproducible problem involving disparate runtime environments (inside
>>> and outside the JVM), executing black-box code on an undisclosed platform
>>> yielding an error message possibly originated by an entity outside the JVM
>>> with no data forthcoming - is that the issue with which you request assistance?
>> 
>> The error message did orginate within the JVM.  I know the problem is
>> a tricky one which is why I specificly worded it to relate to the
>> java.io.File class.
>
>The message originated in the JVM; it doesn't mean that the problem did.
>
>>> Note:  the preceding paragraph implies a path forward, in that a problem statement
>>> illuminates prospective solutions.  In case you missed it, the path forward is
>>> for you to gain visibility into that plethora of unknowns, or sadly report
>>> to your client, as we do to you, that with nothing to go on you have nothing to offer.
>> 
>> Unfortunately I work in the real world.  I often get reported problems
>
>My advice is for the real world.  Duh.
>
>> with inadequate information.  Usually I can work out what is going on
>
>There's a difference between inadequate information and zero information.
>You have given us zero information.

Wrong.  I have given information.

>> even though it's like doing a jigsaw puzzle with most of the pieces
>> missing.  I don't like telling clients that I have nothing to offer in
>> case they tell me the same ;-)
>
>That's why I gave you advice on how to gather more information.
>You have to know /something/ in order to fix the problem, don't you?
>
>>> Visibility involves things like logging.  You do use logging, don't you?
>> 
>> The author of the code did use logging.  It's not sufficiently
>> detailed to be of much use though.
>
>Thus illustrating my point that programmers write terrible logs.

Which programmers?

>I hear your admirable commitment to do some good, but honestly you need a lot more
>information than you have shared here to even begin.  I also hear your rejection
>of my advice to gather such information.  Good luck with that.

I have not rejected your advise, just explained why I had not yet
followed it.  I realize this is not an ideal situation but I have
two choices:
1. To try to do something even without adequate information.
2. Sit on my ass an do nothing until such time as I get more
   information.

I chose (1).  It sounds like you prefer (2).

[toc] | [prev] | [next] | [standalone]

#10529

On 12/5/2011 7:14 AM, Mark wrote:
> Unless the Java class has been badly written we can assume that
> when/if the object is collected then any OS resources that it used
> would also be freed.  Would you agree with this?


Well, the problem here is that not all objects are GC'd in their 
lifetimes.  When the JVM shuts down, no objects are GC'd, the memory 
they occupy is just released.  No finalizers are run.

I've had this happen personally to me.  Forget to close a stream, and 
when the program ends no data is written.  It's still buffered in the 
JVM.  Close your streams and you get data on the disc, like you expected.


> We also see that File does not have a close(), dispose() or similar
> method.  Therefore, if the object does use any OS resources, there is
> no way for the programmer to force them to be released.


The File object represents a path.  It's basically just a String. 
Please stop bringing it up, it's got nothing to do with the issue of OS 
resources.


> On Fri, 2 Dec 2011 14:02:05 -0800 (PST), Lew<lewbloch@gmail.com>
> wrote:
>> 'FileInputStream', but not all 'InputStream' types, and 'FileOutputStream', but
>> not all 'OutputStream' types, do have a 'finalize()' method that might, eventually,
>> release file resources assuming the instances go through two GC cycles, but there's
>> no assurance that that will happen.

> Which, presumably, they have close() methods defined.


Are you sure you're calling close() in all cases?  It sounds like not, 
perhaps you are, but that's the point being made here.


>> Some people believe in Creationism.  It doesn't make their belief valid.
>
> Oh Boy.

He's pointing out that despite references to methods and finalizers, it 
doesn't look like they're being called properly.  We need better 
evidence of what is actually happening to make a better diagnosis of the 
problem.  Everything right now is just guesses and hunches.


> 1. To try to do something even without adequate information.
> 2. Sit on my ass an do nothing until such time as I get more
>     information.
>
> I chose (1).  It sounds like you prefer (2).


"Doing something" should be, imo, making a plan to gather more 
information.  Where can you add logging that will prove, or disprove, 
the current theories about what is happening with the program?

I'm sure there's a unix command to dump resources (like file handles) 
held by each process.  How about dump the out put of that into a file 
before each shell command so we can see who is actually holding on to 
those resources.

Also think about increasing the amount of file handles allowed by the 
JVM or other processes.  The default is actually pretty restrictive and 
designing to stop "abusive" processes.  Really some processes need much 
more.  Don't be afraid to increase it if needed.

[toc] | [prev] | [next] | [standalone]

#10530

On Mon, 05 Dec 2011 07:49:52 -0800, markspace <-@.> wrote:

>On 12/5/2011 7:14 AM, Mark wrote:
>> Unless the Java class has been badly written we can assume that
>> when/if the object is collected then any OS resources that it used
>> would also be freed.  Would you agree with this?
>
>Well, the problem here is that not all objects are GC'd in their 
>lifetimes.  When the JVM shuts down, no objects are GC'd, the memory 
>they occupy is just released.  No finalizers are run.
>
>I've had this happen personally to me.  Forget to close a stream, and 
>when the program ends no data is written.  It's still buffered in the 
>JVM.  Close your streams and you get data on the disc, like you expected.

Which is why I would always explicitly release a resource as soon as
it is not needed, rather than rely on GC.

>> We also see that File does not have a close(), dispose() or similar
>> method.  Therefore, if the object does use any OS resources, there is
>> no way for the programmer to force them to be released.
>
>The File object represents a path.  It's basically just a String. 
>Please stop bringing it up, it's got nothing to do with the issue of OS 
>resources.

I kept bringing it up because I had not had an answer -- until now.

>> On Fri, 2 Dec 2011 14:02:05 -0800 (PST), Lew<lewbloch@gmail.com>
>> wrote:
>>> 'FileInputStream', but not all 'InputStream' types, and 'FileOutputStream', but
>>> not all 'OutputStream' types, do have a 'finalize()' method that might, eventually,
>>> release file resources assuming the instances go through two GC cycles, but there's
>>> no assurance that that will happen.
>
>> Which, presumably, they have close() methods defined.
>
>Are you sure you're calling close() in all cases?  It sounds like not, 
>perhaps you are, but that's the point being made here.

I can't be 100% sure since there is a lot of code but I have been
through it all and cannot find any.  I have seen similar problems to
this before and I have managed to find the culprit without too much
effort.  However one thing I am not clear about is which Java objects
/may/ use or retain OS resources - in this case file handles.

>>> Some people believe in Creationism.  It doesn't make their belief valid.
>>
>> Oh Boy.
>
>He's pointing out that despite references to methods and finalizers, it 
>doesn't look like they're being called properly.  We need better 
>evidence of what is actually happening to make a better diagnosis of the 
>problem.  Everything right now is just guesses and hunches.
>
>> 1. To try to do something even without adequate information.
>> 2. Sit on my ass an do nothing until such time as I get more
>>     information.
>>
>> I chose (1).  It sounds like you prefer (2).
>
>"Doing something" should be, imo, making a plan to gather more 
>information.  Where can you add logging that will prove, or disprove, 
>the current theories about what is happening with the program?

There's lots of things I have already done including adding extra
logging and requesting more information.  However, until I get a reply
to my requests, I don't feel it is appropriate for me to stop.

>I'm sure there's a unix command to dump resources (like file handles) 
>held by each process.  How about dump the out put of that into a file 
>before each shell command so we can see who is actually holding on to 
>those resources.

That is one of the things I have already asked for.

>Also think about increasing the amount of file handles allowed by the 
>JVM or other processes.  The default is actually pretty restrictive and 
>designing to stop "abusive" processes.  Really some processes need much 
>more.  Don't be afraid to increase it if needed.

[toc] | [prev] | [next] | [standalone]

#10535

Mark wrote:
>Lew wrote:
>> Mark wrote:
>>> AFAIK many classes have a close() method to allow any underlying OS
>>> resources to be explicitly freed without needing to wait for the
>>> dispose() method to do this.  If the File method does uses file
>>
>> What 'dispose()' method?  'File' doesn't have one.  'FileOutputStream' doesn't have
>> one.  'FileInputStream' doesn't have one.  None of their ancestor classes up through
>> 'Object' has one.
>>
>>> descriptors then we may assume that these could be left open until the
>>> object is destroyed during GC.
>>
>> No, we cannot assume any such thing, nor can we even assume any given 'File' instance,
>> or any other class instance, will be collected ever.
> 
> Unless the Java class has been badly written we can assume that
> when/if the object is collected then any OS resources that it used
> would also be freed.  Would you agree with this?

Certainly not.

You cannot assume any such thing.

If an object has a 'finalize()' method that does such things, then it will be documented and there will be certain knowledge, not assumptions.

> We also see that File does not have a close(), dispose() or similar
> method.  Therefore, if the object does use any OS resources, there is
> no way for the programmer to force them to be released.

We have already seen that 'File' does use OS resources.

>> 'FileInputStream', but not all 'InputStream' types, and 'FileOutputStream', but
>> not all 'OutputStream' types, do have a 'finalize()' method that might, eventually,
>> release file resources assuming the instances go through two GC cycles, but there's
>> no assurance that that will happen.
> 
> Which, presumably, they have close() methods defined.

Apples and oranges.  I was addressing the question of automatic closure, not overt invocation of the 'close()' method.  The two are orthogonal.  An object can have a non-trivial 'finalize()' method and no 'close()' method, lack override of 'finalize()' but have a 'close()', both override 'finalize()' and have 'close()', or neither.

In the case of an overt 'close()', the 'try...finally' idiom can assure resource release, but does require explicit coding.

In the case of a 'finalize()' that releases resources, you can allow resources to be released automatically, but you run the risk that they never will be.  Furthermore, such a 'finalize()' will increase memory pressure because it delays or prevents garbage collection of objects with such an override.

... [snip] ...

> I have not rejected your advise, just explained why I had not yet
> followed it.  I realize this is not an ideal situation but I have
> two choices:
> 1. To try to do something even without adequate information.
> 2. Sit on my ass an do nothing until such time as I get more
>    information.
> 
> I chose (1).  It sounds like you prefer (2).

I don't prefer either approach, nor have I recommended anything like either one.  I really have no idea how you reached that conclusion.

What I did recommend is "3.", gather more information then act.  It's an example of "Measure Twice, Cut Once".  The problem with taking action without any data is that the action is highly unlikely to help, and if it does help you won't know why.  The problem with sitting on your ass and doing nothing is that you never do anything.  I recommend neither doing nothing nor doing something blindly in hopes that you break nothing.  Rather, do something intentional and tactical with an actual chance of being helpful.

-- 
Lew

[toc] | [prev] | [next] | [standalone]

#10550

On Mon, 5 Dec 2011 15:25:01 -0800 (PST), Lew <lewbloch@gmail.com>
wrote:

>Mark wrote:
>>Lew wrote:
>>> Mark wrote:
>>>> AFAIK many classes have a close() method to allow any underlying OS
>>>> resources to be explicitly freed without needing to wait for the
>>>> dispose() method to do this.  If the File method does uses file
>>>
>>> What 'dispose()' method?  'File' doesn't have one.  'FileOutputStream' doesn't have
>>> one.  'FileInputStream' doesn't have one.  None of their ancestor classes up through
>>> 'Object' has one.
>>>
>>>> descriptors then we may assume that these could be left open until the
>>>> object is destroyed during GC.
>>>
>>> No, we cannot assume any such thing, nor can we even assume any given 'File' instance,
>>> or any other class instance, will be collected ever.
>> 
>> Unless the Java class has been badly written we can assume that
>> when/if the object is collected then any OS resources that it used
>> would also be freed.  Would you agree with this?
>
>Certainly not.
>You cannot assume any such thing.

Then java objects can be collected leaving behind OS resources that
were allocated by that object?  If this is the case then Java is
broken.

>If an object has a 'finalize()' method that does such things, then it will
>be documented and there will be certain knowledge, not assumptions.
>
>> We also see that File does not have a close(), dispose() or similar
>> method.  Therefore, if the object does use any OS resources, there is
>> no way for the programmer to force them to be released.
>
>We have already seen that 'File' does use OS resources.

So there are java methods that use OS resources where you cannot
explicitly free those resources or assume that they will be freed when
collected?  I find this hard to believe.  If this were true then it
would be impossible to write Java applications, using some standard
java classes, /without/ resource leaks.

>>> 'FileInputStream', but not all 'InputStream' types, and 'FileOutputStream', but
>>> not all 'OutputStream' types, do have a 'finalize()' method that might, eventually,
>>> release file resources assuming the instances go through two GC cycles, but there's
>>> no assurance that that will happen.
>> 
>> Which, presumably, they have close() methods defined.
>
>Apples and oranges.  I was addressing the question of automatic closure, not overt
>invocation of the 'close()' method.  The two are orthogonal.  An object can have a
>non-trivial 'finalize()' method and no 'close()' method, lack override of 'finalize()'
>but have a 'close()', both override 'finalize()' and have 'close()', or neither.

It's the "neither" case I am concerned about.

>In the case of an overt 'close()', the 'try...finally' idiom can assure resource
>release, but does require explicit coding.
>
>In the case of a 'finalize()' that releases resources, you can allow resources to be
>released automatically, but you run the risk that they never will be.  Furthermore,
>such a 'finalize()' will increase memory pressure because it delays or prevents
>garbage collection of objects with such an override.

Indeed.

>... [snip] ...
>
>> I have not rejected your advise, just explained why I had not yet
>> followed it.  I realize this is not an ideal situation but I have
>> two choices:
>> 1. To try to do something even without adequate information.
>> 2. Sit on my ass an do nothing until such time as I get more
>>    information.
>> 
>> I chose (1).  It sounds like you prefer (2).
>
>I don't prefer either approach, nor have I recommended anything like either one.
>I really have no idea how you reached that conclusion.
>
>What I did recommend is "3.", gather more information then act.  

But until that information is available, what then?  And by posting
here I was hoping to gather more information about Java's use of OS
resources.

>I recommend neither doing nothing nor doing something blindly in hopes
>that you break nothing.  Rather, do something intentional and tactical with an
>actual chance of being helpful.

Which is exactly what I am attempting to do.  Even when inadequate
information is presented it is often possible to spot obvious coding
errors.  Unfortunately, in this case, there is nothing obvious to me.

[toc] | [prev] | [next] | [standalone]

#10439

On 12/2/2011 11:31 AM, Mark wrote:
> On Fri, 2 Dec 2011 08:17:15 -0800 (PST), Lew<lewbloch@gmail.com>
> wrote:
>> Mark wrote:
>>> Lew wrote:
>>>> Mark wrote:
>>>>> Can a java.io.File object use a OS file descriptor?  I am trying to
>>>>> find the source of a fd leak in a[n] application.
>>>>
>>>> At some point, depending on the operations performed by the 'File' instance,
>>>> there may be a file descriptor involved, and then the 'File' instance certainly
>>>> does use it, at least indirectly via JVM system calls that proxy to OS system
>>>> calls.
>>>>
>>>>  From a Java perspective you should look for unclosed I/O streams/channels and
>>>> packratted 'File' instances rather than file descriptors.
>>>
>>> I've done a code inspection and the streams are all explicitly closed.
>>> There are a number of File objects used and I notice that File does
>>> not have a close() method so we have to rely on GC.
>>
>> If there were a 'close()' method, as there is with streams, it would have
>> nothing to do with GC.  'close()' is for resources (such as file handles).
>> GC is for heap memory.  I only suggested checking for packratted 'File'
>> instances as a foolish guess.  Now that I think about it, it is highly
>> unlikely that unclaimed instances would have anything to do with your issue.
>
> AFAIK many classes have a close() method to allow any underlying OS
> resources to be explicitly freed without needing to wait for the
> dispose() method to do this.  If the File method does uses file
> descriptors then we may assume that these could be left open until the
> object is destroyed during GC.

That will only happen during GC if there is a finalizer.

And the usage of finalizers to free resources usually
raises a lot of red flags!

Arne

[toc] | [prev] | [next] | [standalone]

#10460

On Fri, 02 Dec 2011 15:55:17 +0000, Mark
<i@dontgetlotsofspamanymore.net> wrote, quoted or indirectly quoted
someone who said :

>I've done a code inspection and the streams are all explicitly closed.
>There are a number of File objects used and I notice that File does
>not have a close() method so we have to rely on GC.

Files are just a wrapper around a file name to let you do
manipulations on the name, You can also do delete and lastmodified
which do not open the file.  If you have dangling open files, it must
involve something with a close method.

Surely some profiler will tell you.  One of the sysinternals utilities
will let you track open/close at the OS level.

see http://mindprod.com/jgloss/profiler.html
http://mindprod.com/jgloss/systeminternals.html

Failing that you will have to instrument your code by calling a logger
at every open and close.

-- 
Roedy Green Canadian Mind Products
http://mindprod.com
For me, the appeal of computer programming is that
even though I am quite a klutz,
I can still produce something, in a sense
perfect, because the computer gives me as many
chances as I please to get it right.
 

[toc] | [prev] | [next] | [standalone]

#10429

On 02/12/2011 16:25, Lew wrote:
> Mark wrote:
>> Can a java.io.File object use a OS file descriptor?  I am trying to
>> find the source of a fd leak in a[n] application.
>
> At some point, depending on the operations performed by the 'File' instance, there may be a file descriptor involved, and then the 'File' instance certainly does use it, at least indirectly via JVM system calls that proxy to OS system calls.

It was my understanding that java.io.File instances were merely 
representations of pathnames, and are neither closed nor opened, just 
places and names where files could exist or not.

It was also my understanding that file descriptors were representations 
of opened files.

--
Mayeul

[toc] | [prev] | [next] | [standalone]

Page 1 of 2  [1] 2  Next page →

Back to top | Article view | comp.lang.java.programmer

csiph-web