Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.java.programmer > #34438 > unrolled thread

Java program "debugging" itself...

Back to article view | Back to comp.lang.java.programmer

Contents

  Java program "debugging" itself... Andreas Leitgeb <avl@logic.at> - 2016-06-02 13:36 +0000
    Re: Java program "debugging" itself... Martin Gregorie <martin@address-in-sig.invalid> - 2016-06-02 14:17 +0000
      Re: Java program "debugging" itself... Andreas Leitgeb <avl@logic.at> - 2016-06-02 15:06 +0000
        Re: Java program "debugging" itself... Eric Sosman <esosman@comcast-dot-net.invalid> - 2016-06-02 11:14 -0400
          Re: Java program "debugging" itself... Andreas Leitgeb <avl@logic.at> - 2016-06-02 15:43 +0000
    Re: Java program "debugging" itself... "comp.lang.coffee" <kirandesai84@gmail.com> - 2016-06-02 07:21 -0700
      Re: Java program "debugging" itself... Eric Sosman <esosman@comcast-dot-net.invalid> - 2016-06-02 10:38 -0400
        Re: Java program "debugging" itself... Robert Klemme <shortcutter@googlemail.com> - 2016-06-02 22:24 +0200
      Re: Java program "debugging" itself... Andreas Leitgeb <avl@logic.at> - 2016-06-02 15:28 +0000
        Re: Java program "debugging" itself... Martin Gregorie <martin@address-in-sig.invalid> - 2016-06-02 22:22 +0000
          Re: Java program "debugging" itself... Andreas Leitgeb <avl@logic.at> - 2016-06-03 15:58 +0000
            Re: Java program "debugging" itself... Wanja Gayk <brixomatic@yahoo.com> - 2016-06-04 21:32 +0200

#34438 — Java program "debugging" itself...

My application is a server that runs separate tasks as 
requested by clients.

Rarely it happens, that one of the tasks goes awry, due
to some bug in the task's code itself.  Latest incident
was a loop that accidentally didn't terminate but grew a 
StringBuilder's buffer beyond any bounds.

Given that there exist some natural "constraints" on these
tasks, like they would *never ever* allocate an array of
more than a million elements (yes, that's a rather low
upper bound), then I'd like to use some debugging-API to
request that any thread trying to "new" any array of
size >= 1,000,000 ought to get e.g. an OutOfMemoryError -
regardless of actually available heap-space. This of
course would include "new" statements happening from
the standard library (like StringBuilder's .append(...) ).

Does anyone know, whether any VMs offer any (maybe non-
standard) API for applications to "watch" themselves?

The whole point is that any task may fail, but the show
(i.e. the server) must go on...

[toc] | [next] | [standalone]

#34439

On Thu, 02 Jun 2016 13:36:37 +0000, Andreas Leitgeb wrote:

> My application is a server that runs separate tasks as requested by
> clients.
> 
> Rarely it happens, that one of the tasks goes awry, due to some bug in
> the task's code itself.  Latest incident was a loop that accidentally
> didn't terminate but grew a StringBuilder's buffer beyond any bounds.
>
Are these tasks running in their own address space or are they sharing an 
address space with your server and its other children?

If the former is the case, can you use the -Xmx command line option to 
trap and kill off the misbehaving task? 

In the past I've had the opposite effect when bulk loading data into a 
Postgres database on a Linux system: my Java application, when run on a 
smaller system than I have now, kept being killed off, initially when it 
exceeded the -Xmx default value and then several more times until I 
finally got an -Xmx setting that was big enough to let it run to 
completion. My current system has 8 times the memory and the so the 
default -Xmx setting is now big enough not to interfere. 


-- 
martin@   | Martin Gregorie
gregorie. | Essex, UK
org       |

[toc] | [prev] | [next] | [standalone]

#34442

Martin Gregorie <martin@address-in-sig.invalid> wrote:
> On Thu, 02 Jun 2016 13:36:37 +0000, Andreas Leitgeb wrote:
>> My application is a server that runs separate tasks as requested by
>> clients.
>> Rarely it happens, that one of the tasks goes awry, due to some bug in
>> the task's code itself.  Latest incident was a loop that accidentally
>> didn't terminate but grew a StringBuilder's buffer beyond any bounds.
> Are these tasks running in their own address space or are they sharing an 
> address space with your server and its other children?

Well, all is running in a single VM, and it has to be that way.

Most of the tasks are "quick-shots" (that complete within fractions
of seconds) and interactive (dialog) applications.  If I started them
in their own VM, these would take up to 10 seconds to finish or to
become ready. (actual measurements from earlier stages of the project)

The tasks however don't share any business data among each other, so
if one task allocates 10 gigabytes of chunks on heap, then no other
task will ever use those chunks.

[toc] | [prev] | [next] | [standalone]

#34443

On 6/2/2016 11:06 AM, Andreas Leitgeb wrote:
> Martin Gregorie <martin@address-in-sig.invalid> wrote:
>> On Thu, 02 Jun 2016 13:36:37 +0000, Andreas Leitgeb wrote:
>>> My application is a server that runs separate tasks as requested by
>>> clients.
>>> Rarely it happens, that one of the tasks goes awry, due to some bug in
>>> the task's code itself.  Latest incident was a loop that accidentally
>>> didn't terminate but grew a StringBuilder's buffer beyond any bounds.
>> Are these tasks running in their own address space or are they sharing an
>> address space with your server and its other children?
>
> Well, all is running in a single VM, and it has to be that way.
>
> Most of the tasks are "quick-shots" (that complete within fractions
> of seconds) and interactive (dialog) applications.  If I started them
> in their own VM, these would take up to 10 seconds to finish or to
> become ready. (actual measurements from earlier stages of the project)

     As a hybrid approach, could you use a suite of N long-running JVM's,
each executing one task at a time but executing many in succession?  In
normal operation you'd "amortize" the JVM startup cost over the many
tasks it would handle in its lifetime, but still be able to kill off a
task that misbehaved (by killing, and presumably replacing, its JVM).

     Sounds like a lot of work, but I can't think of a way within one JVM
to insulate the "benign" tasks from a "malignant" one.

-- 
esosman@comcast-dot-net.invalid
"Don't be afraid of work. Make work afraid of you." -- TLM

[toc] | [prev] | [next] | [standalone]

#34445

Eric Sosman <esosman@comcast-dot-net.invalid> wrote:
> On 6/2/2016 11:06 AM, Andreas Leitgeb wrote:
>> Martin Gregorie <martin@address-in-sig.invalid> wrote:
>>> On Thu, 02 Jun 2016 13:36:37 +0000, Andreas Leitgeb wrote:
>>>> My application is a server that runs separate tasks as requested by
>>>> clients.
>>>> Rarely it happens, that one of the tasks goes awry, due to some bug in
>>>> the task's code itself.  Latest incident was a loop that accidentally
>>>> didn't terminate but grew a StringBuilder's buffer beyond any bounds.
>>> Are these tasks running in their own address space or are they sharing an
>>> address space with your server and its other children?
>> Well, all is running in a single VM, and it has to be that way.
>> Most of the tasks are "quick-shots" (that complete within fractions
>> of seconds) and interactive (dialog) applications.  If I started them
>> in their own VM, these would take up to 10 seconds to finish or to
>> become ready. (actual measurements from earlier stages of the project)
>      As a hybrid approach, could you use a suite of N long-running JVM's,
> each executing one task at a time but executing many in succession?

We have a similar workaround already: a VM can be detached from its
controlling channel (where it gets the requests for new tasks), and
keep running to finish its already running tasks. A new VM is then
started to accept and handle new requests.

After some hours, all legal tasks in the backgrounded Server will
have finished and then the VM is manually killed by OS means.

This however requires some monitoring of the VM and manual intervention
if one VM behaves unusually.

That's why I'd hope to have certain types of "rogue" tasks swat themselves
automatically.

[toc] | [prev] | [next] | [standalone]

#34440

On Thursday, June 2, 2016 at 8:36:51 AM UTC-5, Andreas Leitgeb wrote:
> My application is a server that runs separate tasks as 
> requested by clients.
> 
> Rarely it happens, that one of the tasks goes awry, due
> to some bug in the task's code itself.  Latest incident
> was a loop that accidentally didn't terminate but grew a 
> StringBuilder's buffer beyond any bounds.
> 
> Given that there exist some natural "constraints" on these
> tasks, like they would *never ever* allocate an array of
> more than a million elements (yes, that's a rather low
> upper bound), then I'd like to use some debugging-API to
> request that any thread trying to "new" any array of
> size >= 1,000,000 ought to get e.g. an OutOfMemoryError -
> regardless of actually available heap-space. This of
> course would include "new" statements happening from
> the standard library (like StringBuilder's .append(...) ).
> 
> Does anyone know, whether any VMs offer any (maybe non-
> standard) API for applications to "watch" themselves?
> 
> The whole point is that any task may fail, but the show
> (i.e. the server) must go on...

It looks to me like you could use custom exception handling to solve your problem. You can set the limit configurable so you can update it as required. 

[toc] | [prev] | [next] | [standalone]

#34441

On 6/2/2016 10:21 AM, comp.lang.coffee wrote:
> On Thursday, June 2, 2016 at 8:36:51 AM UTC-5, Andreas Leitgeb wrote:
>> My application is a server that runs separate tasks as
>> requested by clients.
>>
>> Rarely it happens, that one of the tasks goes awry, due
>> to some bug in the task's code itself.  Latest incident
>> was a loop that accidentally didn't terminate but grew a
>> StringBuilder's buffer beyond any bounds.
>>
>> Given that there exist some natural "constraints" on these
>> tasks, like they would *never ever* allocate an array of
>> more than a million elements (yes, that's a rather low
>> upper bound), then I'd like to use some debugging-API to
>> request that any thread trying to "new" any array of
>> size >= 1,000,000 ought to get e.g. an OutOfMemoryError -
>> regardless of actually available heap-space. This of
>> course would include "new" statements happening from
>> the standard library (like StringBuilder's .append(...) ).
>>
>> Does anyone know, whether any VMs offer any (maybe non-
>> standard) API for applications to "watch" themselves?
>>
>> The whole point is that any task may fail, but the show
>> (i.e. the server) must go on...
>
> It looks to me like you could use custom exception handling to solve your problem. You can set the limit configurable so you can update it as required.

     If you suggest he should catch OutOfMemoryException and stop the
thread that caught it, I don't think that will work.  (If you meant
something else, just ignore this ...)

     The problem is a lot like that of Linux' OOM Killer: The task that
hits the limit is not necessarily the task at fault.  If a runaway task
gobbles up almost all the memory and then a well-behaved task gets an
OutOfMemoryException on something innocuous like `new int[10]', the
exception is thrown to the "wrong" thread: Not the runaway, but the
innocent bystander.

     The traditional approach to this sort of problem is to run each
task in its own JVM.  I'm sure Andreas is aware of this, and I'm sure
he'd do that if it made sense (it might not if, for example, most of
the tasks are short-lived).  Since he's asking for another means of
attack, I surmise that something in his situation makes the multi-JVM
approach undesirable.

     Unfortunately, I know of no way to "contain the damage" when one
of several tasks in the same JVM decides to go off the rails ...

-- 
esosman@comcast-dot-net.invalid
"Don't be afraid of work. Make work afraid of you." -- TLM

[toc] | [prev] | [next] | [standalone]

#34446

On 02.06.2016 16:38, Eric Sosman wrote:

>      The problem is a lot like that of Linux' OOM Killer: The task that
> hits the limit is not necessarily the task at fault.  If a runaway task
> gobbles up almost all the memory and then a well-behaved task gets an
> OutOfMemoryException on something innocuous like `new int[10]', the
> exception is thrown to the "wrong" thread: Not the runaway, but the
> innocent bystander.

I think this hints straight at an important underlying issue - 
regardless of whether the option exists that Andreas has asked for: how 
do you determine the "culprit"?  Eric has illustrated one example, 
here's another one: one thread allocates tens of thousands of int[100] 
while another thread allocates a single char[10000] hitting a memory 
limit (or worse, OOM because of fragmentation).

Basically it starts with, you would have to keep track of the overall 
resource consumption of a task or thread.  Then define criteria for 
termination and then you need to find a way to implement that.  One 
approach that comes to mind is JMX: a process can connect to itself and 
monitor its own activity (btw. "monitor" is a much better term than 
"debug" for this thread's title).

Kind regards

	robert


-- 
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/

[toc] | [prev] | [next] | [standalone]

#34444

comp.lang.coffee <kirandesai84@gmail.com> wrote:
> On Thursday, June 2, 2016 at 8:36:51 AM UTC-5, Andreas Leitgeb wrote:
>> My application is a server that runs separate tasks as 
>> requested by clients.
>> 
>> Rarely it happens, that one of the tasks goes awry, due
>> to some bug in the task's code itself.  Latest incident
>> was a loop that accidentally didn't terminate but grew a 
>> StringBuilder's buffer beyond any bounds.
>> 
>> Given that there exist some natural "constraints" on these
>> tasks, like they would *never ever* allocate an array of
>> more than a million elements (yes, that's a rather low
>> upper bound), then I'd like to use some debugging-API to
>> request that any thread trying to "new" any array of
>> size >= 1,000,000 ought to get e.g. an OutOfMemoryError -
>> regardless of actually available heap-space. This of
>> course would include "new" statements happening from
>> the standard library (like StringBuilder's .append(...) ).
>> 
>> Does anyone know, whether any VMs offer any (maybe non-
>> standard) API for applications to "watch" themselves?
>> 
>> The whole point is that any task may fail, but the show
>> (i.e. the server) must go on...
>
> It looks to me like you could use custom exception handling
> to solve your problem. You can set the limit configurable so
> you can update it as required. 

I'm afraid, I don't understand.

The whole VM has a heap of about 32 gigabytes.  A single task
creating a new char[1000000] would hardly ever fill the heap
or *normally* trigger any exception. But if that thread is left
running, it will most likely continue to allocate even larger
arrays (e.g.: loop around aStringBuilder.append(...)), and
eventually later the heap will actually fill up, possibly
causing collateral trouble. That's what I try to avoid.

Based on the design of the tasks, I know for sure that if such
a big array gets allocated by one of the tasks, then this is a
safe indicator for a bug.

A single task may (e.g.) legally allocate an 65536-sized array
of char-arrays each sized 500,000 - but if it ever allocates a
single array of 1,000,000 elements, then this only ever happens
in consequence of a bug, so the task doing that should be killed
as soon as possible (before it has a chance to really eat up all
the heap).

How can I set/configure limits for array size to cause custom
exceptions to be thrown for "threspassing" threads?

[toc] | [prev] | [next] | [standalone]

#34447

On Thu, 02 Jun 2016 15:28:56 +0000, Andreas Leitgeb wrote:

> How can I set/configure limits for array size to cause custom exceptions
> to be thrown for "threspassing" threads?

Who writes these tasks? IOW, is there any way you could force all memory 
acquisition and release by these tasks to go through your main process 
which could then keep track of memory used by each task and kill them if 
they break agreed quotas? 

I know this is a bit of a faff in Java (it would be easy to handle in C 
with just the one malloc()/free() mechanism), but it is a way to enforce 
memory use limits on multiple tasks/threads in a single JVM. Its biggest 
drawback is probably that it relies on all programmers obeying the rules.


-- 
martin@   | Martin Gregorie
gregorie. | Essex, UK
org       |

[toc] | [prev] | [next] | [standalone]

#34456

Martin Gregorie <martin@address-in-sig.invalid> wrote:
> On Thu, 02 Jun 2016 15:28:56 +0000, Andreas Leitgeb wrote:
>> How can I set/configure limits for array size to cause custom exceptions
>> to be thrown for "threspassing" threads?
> Who writes these tasks? IOW, is there any way you could force all memory 
> acquisition and release by these tasks to go through your main process 
> which could then keep track of memory used by each task and kill them if 
> they break agreed quotas? 

The tasks are auto-converted from another language that had no concept
of a heap, and its static and local variables were quite limited in size.

The converted tasks themselves "cannot" leak nor pack-rat memory,
but my own methods (that make up the library) may have bugs.

As the (converted) application is in productive use already for
about a year now, the remaining bugs seem to have become pretty rare.

Changing all array creations to method calls would be a pretty big
change with a non-trivial risk of introducing new bugs.  And I'd
have to wrap StringBuilder and many Collection classes, too.

That's why I hoped for some "meta-API" of the JVM, that would allow
me to control maximum allowable array-sizes (and maybe other settings
that I haven't even yet dreamt of), which would allow me to set up a
custom advanced safety net beyond the standard one within the JVM.

[toc] | [prev] | [next] | [standalone]

#34470

In article <slrnnl3aa2.456.avl@login.logic.tuwien.ac.at>, Andreas 
Leitgeb (avl@logic.at) says...

> The tasks are auto-converted from another language that had no concept
> of a heap, and its static and local variables were quite limited in size.
> 
> The converted tasks themselves "cannot" leak nor pack-rat memory,
> but my own methods (that make up the library) may have bugs.
> 
> As the (converted) application is in productive use already for
> about a year now, the remaining bugs seem to have become pretty rare.
> 
> Changing all array creations to method calls would be a pretty big
> change with a non-trivial risk of introducing new bugs.  And I'd
> have to wrap StringBuilder and many Collection classes, too.
> 
> That's why I hoped for some "meta-API" of the JVM, that would allow
> me to control maximum allowable array-sizes (and maybe other settings
> that I haven't even yet dreamt of), which would allow me to set up a
> custom advanced safety net beyond the standard one within the JVM.

Probably you could do this:

1. use an MBean to trigger a warning mechanism, before reaching memory 
limits:

http://www.inoneo.com/en/blog/8/java/monitoring-jvm-memory-usage-inside-
a-java-application

1a: have this warning mechanism call "jcmd <pid> GC.class_histogram" to 
dump into a file how many instances of each class are currently on the 
heap (make sure you set a timed limit when to trigger this, you don't 
want to fill your HD with dumps, and you don't want to dump too often).

or 

1b: with a little bit use of deprecated APIs, (Java9 will hate you), 
dump an estimate of the amount of bytes each thread allocated:

final com.sun.management.ThreadMXBean threadMXBean = 
(com.sun.management.ThreadMXBean) ManagementFactory.getThreadMXBean();
final long[] threadIDs = threadMXBean.getAllThreadIds();
final ThreadInfo[] threadInfos = threadMXBean.getThreadInfo(threadIDs);

final Map<ThreadInfo, Long> threadToMem = new HashMap<>();
Arrays.stream(threadInfos)
 .forEach(
  info -> threadToMem.put(info, threadMXBean.getThreadAllocatedBytes
(info.getThreadId())));

final TreeMap<ThreadInfo, Long> sortedMap 
 = new TreeMap<>((i1, i2) -> -Long.compare(threadToMem.get(i1), 
threadToMem.get(i2)));

sortedMap.putAll(threadToMem);

//dump info:
sortedMap
 .forEach((i, m) -> System.out.println(m + "\t" + i.getThreadName()));


Note that the number of bytes will not decrease on garbage collection, 
it will increase monotonically. Still, depending on how the program 
works, it may still provide a good hint which thread is excessivelly 
filling up memory.

See also: 
https://docs.oracle.com/javase/7/docs/jre/api/management/extension/com/s
un/management/ThreadMXBean.html


Kind regards,
Wanja


-- 
..Alesi's problem was that the back of the car was jumping up and down 
dangerously - and I can assure you from having been teammate to 
Jean Alesi and knowing what kind of cars that he can pull up with, 
when Jean Alesi says that a car is dangerous - it is. [Jonathan Palmer]

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.java.programmer

csiph-web