Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > microsoft.public.scripting.vbscript > #12281 > unrolled thread

using wshell.exec causes lockup while reading the output - what to do ?

Back to article view | Back to microsoft.public.scripting.vbscript

Contents

  using wshell.exec causes lockup while reading the output - what to do ? "R.Wieser" <address@not.available> - 2020-06-17 14:32 +0200
    Re: using wshell.exec causes lockup while reading the output - what to do ? "Mayayana" <mayayana@invalid.nospam> - 2020-06-17 08:48 -0400
    Re: using wshell.exec causes lockup while reading the output - what to do ? JJ <jj4public@vfemail.net> - 2020-06-17 22:12 +0700
      Re: using wshell.exec causes lockup while reading the output - what to do ? "R.Wieser" <address@not.available> - 2020-06-17 18:51 +0200
        Re: using wshell.exec causes lockup while reading the output - what to do ? JJ <jj4public@vfemail.net> - 2020-06-18 19:42 +0700
          Re: using wshell.exec causes lockup while reading the output - what to do ? "R.Wieser" <address@not.available> - 2020-06-18 17:26 +0200
            Re: using wshell.exec causes lockup while reading the output - what to do ? JJ <jj4public@vfemail.net> - 2020-06-19 21:16 +0700
              Re: using wshell.exec causes lockup while reading the output - what to do ? "R.Wieser" <address@not.available> - 2020-06-19 19:08 +0200
                Re: using wshell.exec causes lockup while reading the output - what to do ? JJ <jj4public@vfemail.net> - 2020-06-20 22:40 +0700
                  Re: using wshell.exec causes lockup while reading the output - what to do ? "R.Wieser" <address@not.available> - 2020-06-20 18:56 +0200
                    Re: using wshell.exec causes lockup while reading the output - what to do ? JJ <jj4public@vfemail.net> - 2020-06-21 12:33 +0700
                      Re: using wshell.exec causes lockup while reading the output - what to do ? "R.Wieser" <address@not.available> - 2020-06-21 08:53 +0200
                        Re: using wshell.exec causes lockup while reading the output - what to do ? JJ <jj4public@vfemail.net> - 2020-06-21 20:58 +0700
                          Re: using wshell.exec causes lockup while reading the output - what to do ? "R.Wieser" <address@not.available> - 2020-06-21 17:46 +0200

#12281 — using wshell.exec causes lockup while reading the output - what to do ?

Hello all

I'm running a console-based windows program, and need to capture its output. 
When I try to use the "exec" method than, nomatter what I try, something 
locks up.

My code is basically this:

Set oExec = WshShell.Exec(sCmd)
Do While oExec.Status = 0
     WScript.Sleep 100
     wscript.echo oExec.stdout.readall
Loop

As the program also outputs on stderr I've changed the above reading of 
stdout with

if not oExec.stdout.atendofstream then wscript.echo oExec.stdout.read(1000)
if not oExec.stderr.atendofstream then wscript.echo oExec.stderr.read(1000)

But no matter what I've tried (small buffers, large buffers), at some point 
in the output it locks up. :-(

Apart from switching to the "run" method*, what can I do ?

*I've got the "run" method working.  But the output capturing needs the use 
of "cmd /e ", which in turn causes my screen to go black a few times 
(mutiple calls) with no (progress) output on it.  Not nice.

Regards,
Rudy Wieser

[toc] | [next] | [standalone]

#12282

"R.Wieser" <address@not.available> wrote

| I'm running a console-based windows program, and need to capture its 
output.
| When I try to use the "exec" method than, nomatter what I try, something
| locks up.
|
| My code is basically this:
|
| Set oExec = WshShell.Exec(sCmd)
| Do While oExec.Status = 0
|     WScript.Sleep 100
|     wscript.echo oExec.stdout.readall
| Loop
|
| As the program also outputs on stderr I've changed the above reading of
| stdout with
|
| if not oExec.stdout.atendofstream then wscript.echo 
oExec.stdout.read(1000)
| if not oExec.stderr.atendofstream then wscript.echo 
oExec.stderr.read(1000)
|
| But no matter what I've tried (small buffers, large buffers), at some 
point
| in the output it locks up. :-(

  I'm not sure I have an answer, but this is a function
I've also had trouble with. I wrote an HTA to wrap
youtube-dl and would like to get it to print an update
to the HTA window as it progresses. But no matter what
I do, I can't get it to update the HTA UI.

  However, I've never seen it lock up. For what it's worth,
this is my current code:

'......
Set ObjExec = SH.Exec(s1)
  s1 = ""
  While (ObjExec.Status = 0)
    SleepEx 100
    s1 = s1 & ObjExec.StdOut.ReadAll
    s2 = s2 & ObjExec.StdErr.ReadAll
   TReport.value = s1 & vbCrLf & s2
   DoEvents
  Wend

   s1 = s1 & ObjExec.StdOut.ReadAll
   s2 = s2 & ObjExec.StdErr.ReadAll
  TReport.value = s1 & vbCrLf & s2

     Set ObjExec = Nothing

'......

Sub SleepEx(Lms)
  Dim Ret
  Ret = SH2.Run("sleeper.vbs " & Lms, , True)
End Sub

Sub DoEvents()
    SH.run "%comspec% /c exit", 0, True
 End Sub

'....

Sleeper.vbs is just this:

Dim Arg
  on error resume next
 Arg = WScript.Arguments(0)
 wscript.sleep Arg

SleepEx and DoEvents were added to try to make
the HTA update during the process, but it doesn't
work. Only after the whole thing is finished does
the command window disappear and the total output
text appears in the window. Would such pauses help
you? I don't know. Maybe it depends on the executable?

  I have seen a lockup occasionally with youtube-dl.
I've assumed it's a problem with the download itself.
Maybe it got interrupted and youtube-dl couldn't
continue. I don't know. When I see that downloading
has stopped I just run the command again and it picks
up where it left off. But that problem is intermittent.

[toc] | [prev] | [next] | [standalone]

#12283

On Wed, 17 Jun 2020 14:32:35 +0200, R.Wieser wrote:
> Hello all
> 
> I'm running a console-based windows program, and need to capture its output. 
> When I try to use the "exec" method than, nomatter what I try, something 
> locks up.
> 
> My code is basically this:
> 
> Set oExec = WshShell.Exec(sCmd)
> Do While oExec.Status = 0
>      WScript.Sleep 100
>      wscript.echo oExec.stdout.readall
> Loop
> 
> As the program also outputs on stderr I've changed the above reading of 
> stdout with
> 
> if not oExec.stdout.atendofstream then wscript.echo oExec.stdout.read(1000)
> if not oExec.stderr.atendofstream then wscript.echo oExec.stderr.read(1000)
> 
> But no matter what I've tried (small buffers, large buffers), at some point 
> in the output it locks up. :-(
> 
> Apart from switching to the "run" method*, what can I do ?
> 
> *I've got the "run" method working.  But the output capturing needs the use 
> of "cmd /e ", which in turn causes my screen to go black a few times 
> (mutiple calls) with no (progress) output on it.  Not nice.
> 
> Regards,
> Rudy Wieser

When any handle is not a file, TextStream object can't really detect the
presence of data in that handle. Thus, AtEndOfStream() or AtEndOfLine()
won't return unless there's data in the handle. This also applies to all
Read and Skip methods, because they seem to use the same function to detect
presence of data in the handle.

[toc] | [prev] | [next] | [standalone]

#12284

JJ,

> When any handle is not a file, TextStream object can't really detect
> the presence of data in that handle. Thus, AtEndOfStream() or
> AtEndOfLine() won't return unless there's data in the handle.

That should not be a problem.  As long as the program that is exec-ed runs 
it should be generating output (in a seperate thread I presume), and the 
buffer should eventually gain some new contents.   But no, I can see the 
output stops somewhere rather early, nowhere near to the end.

Regardless, do you have any suggestions to how to solve it/what to try ? 
Or should I conclude that trying to grab output from an "exec" method is 
undependable, and thus simply a waste of time ?

Regards,
Rudy Wieser

[toc] | [prev] | [next] | [standalone]

#12285

On Wed, 17 Jun 2020 18:51:41 +0200, R.Wieser wrote:
> 
> That should not be a problem.  As long as the program that is exec-ed runs 
> it should be generating output (in a seperate thread I presume), and the 
> buffer should eventually gain some new contents.   But no, I can see the 
> output stops somewhere rather early, nowhere near to the end.

It doesn't matter whether the program output/error is written by multithread
or not, because the problem lies within the TextStream object's methods.

> Regardless, do you have any suggestions to how to solve it/what to try ? 

When the input is a character device, instead of a file, don't use counted
read operation unless you know exactly when the program stops writing data
into a standard handle. That would produce a deadlock unless the program
generate at least the same length of data being requested to read.

In your code, you have these lines:

if not oExec.stdout.atendofstream then wscript.echo oExec.stdout.read(1000)
if not oExec.stderr.atendofstream then wscript.echo oExec.stderr.read(1000)

If the program writes some lines totalling 500 bytes into standard output,
the script will be stuck within `oExec.stdout.read(1000)` because it's
expecting 500 more bytes of data.

It'be better to read the program output line by line like below.

  do while not oExec.stdout.atendofstream
    wscript.echo oExec.stdout.readline
  loop

> Or should I conclude that trying to grab output from an "exec" method is 
> undependable, and thus simply a waste of time ?

What's wrong with that? I use it myself to parse various programs' output,
as well as input commands into the executed programs.

[toc] | [prev] | [next] | [standalone]

#12286

JJ,

> If the program writes some lines totalling 500 bytes into standard output,
> the script will be stuck within `oExec.stdout.read(1000)` because it's
> expecting 500 more bytes of data.
>
> It'be better to read the program output line by line like below.
>
>  do while not oExec.stdout.atendofstream
>    wscript.echo oExec.stdout.readline
>  loop

Well, that doesn't quite work either: While the programs stdout does (in 
this case!) emit lines, the stderr emits a long stream which never 
terminates.  Trying to read a line there directly blocks the whole thing. 
And yes, I did precede it with an "atendofstream" check. :-)

>> Or should I conclude that trying to grab output from an "exec" method is
>> undependable, and thus simply a waste of time ?
>
> What's wrong with that?

Whut ?    You're not serious, are you ?

Regards,
Rudy Wieser

[toc] | [prev] | [next] | [standalone]

#12287

On Thu, 18 Jun 2020 17:26:48 +0200, R.Wieser wrote:
> 
> ...the stderr emits a long stream which never terminates.

That's a very important information which should have been mentioned in the
initial post. If a stream never terminates - assuming that it never emit an
EOL (CRLF/LF), ReadLine() will never return. So counted read is the only way
to read the stream.

However, you will need to know when the stream data ends. You need to make
sure the code is not reading more bytes than what's available. Otherwise,
it'll get stuck.

If the end of stream data is unknown and is unterminated, then I suggest
redirecting the standard error to a file instead, using CMD. Exec() and
TextStream in this case, can't do the job. AtEndOfStream() will be useless
because it will be stuck if a stream is not a file and has no data.

>>> Or should I conclude that trying to grab output from an "exec" method is
>>> undependable, and thus simply a waste of time ?
>>
>> What's wrong with that?
> 
> Whut ?    You're not serious, are you ?

Sorry. That wasn't said correctly. What I meant is that I use TextStream
too. It's undependable, yes sometimes. But it's not a waste of time, because
it's all we got in WSH. That being said, it can be a waste of time,
depending of the situation due to TextStream limitations.

[toc] | [prev] | [next] | [standalone]

#12288

JJ,

>> ...the stderr emits a long stream which never terminates.
>
> That's a very important information which should have been
> mentioned in the initial post.

If I would have known it was important I probably would have.

> So counted read is the only way to read the stream.
>
> However, you will need to know when the stream data ends.
> You need to make sure the code is not reading more bytes
> than what's available.

Sounds resonable.  So, any idea how I can ask a stream how many 
bytes/characters it has currently buffered ?  :-)

> If the end of stream data is unknown and is unterminated,
> then I suggest redirecting the standard error to a file instead,
> using CMD.

I already got that to work, but it has got its own drawbacks.  Hence my 
attempt to use "exec".

Regards,
Rudy Wieser

P.s.
Besides having to use output chunked into lines, according to the below link 
those lines may not be longer than the internal (pipe)buffer size - which is 
just a measily 4K (no idea if that is in bytes or chars).

https://groups.google.com/d/topic/microsoft.public.scripting.wsh/kIvQsqxSkSk

[toc] | [prev] | [next] | [standalone]

#12289

On Fri, 19 Jun 2020 19:08:39 +0200, R.Wieser wrote:
> 
> Sounds resonable.  So, any idea how I can ask a stream how many 
> bytes/characters it has currently buffered ?  :-)

That's the main problem I've been trying to tell you. TextStream simply
isn't capable enough when the stream is a standard handle which is a
character device. It at least need a help from an outside tool/COM.

> I already got that to work, but it has got its own drawbacks.  Hence my 
> attempt to use "exec".

I'm guessing that you also need to inspect the outputted data right away,
instead of waiting until the program has terminated?

> P.s.
> Besides having to use output chunked into lines, according to the below link 
> those lines may not be longer than the internal (pipe)buffer size - which is 
> just a measily 4K (no idea if that is in bytes or chars).
> 
> https://groups.google.com/d/topic/microsoft.public.scripting.wsh/kIvQsqxSkSk

There's that limit, yes. But it's not what causing TextStream to get stuck.

[toc] | [prev] | [next] | [standalone]

#12290

JJ,

> That's the main problem I've been trying to tell you. TextStream
> simply isn't capable enough when the stream is a standard handle
> which is a character device

So, I /can't/ check if its empty, I /can't/ ask how full it is, and its 
"read data" methods are all blocking ones.

The only way "exec" with output capturing can work is either with having 
just a single stream (on which the "readall" method would probably work), or 
when having two streams that /both/ emit lines of data (so that the 
"readline" method can be used), with both lines being shorter than a buffers 
worth.

That strongly limits the "exec" methods use cases.

Wait ... Nope, having both emit lines will still cause deadlocks.  Just 
imagine one stream getting shorter lines than the other.  As you can't empty 
a stream (trying to do so will cause a blocking situation) and you can't 
skip reading a stream when it doesn't have any data (checking for that also 
blocks), the buffer getting the longer lines will eventually fill up and 
block.

I really start to hate MS.   :-(

> I'm guessing that you also need to inspect the outputted data
> right away, instead of waiting until the program has terminated?

The stderr emits a progress counter for the screen (output count and a CR, 
the new value overwriting the last), stdout is supposed to be written to a 
file.

> There's that limit, yes. But it's not what causing TextStream to
> get stuck.

I think I understood the bit about blocking reads.  The buffersize problem 
goes, when using the "readline" method, ontop of that.   Or did I mis 
something ?

Regards,
Rudy Wieser

[toc] | [prev] | [next] | [standalone]

#12291

On Sat, 20 Jun 2020 18:56:24 +0200, R.Wieser wrote:
> 
> I really start to hate MS.   :-(

Can't live with it, and but can't throw it away either, right?

> The stderr emits a progress counter for the screen (output count and a CR, 
> the new value overwriting the last), stdout is supposed to be written to a 
> file.

If you need to parse the counter, you can use counted read one character at
a time until CR is recevied, then parse the collected counter characters.
Repeat that to process the next counter. That should take care of the
counter output processing without getting a deadlock.

Now the problem is knowing when the couter stops being updated. So, what
data is emitted to the standard error the program has completed its
processing? i.e. the data emitted after the last counter. Whatever that is,
it should be treated as a signal to stop the loop for processing the counter
output. If no more data is emitted to the standard error and the programs
simply ends, then there's no need to ready anything any more. Just exit the
loop.

> I think I understood the bit about blocking reads.  The buffersize problem 
> goes, when using the "readline" method, ontop of that.   Or did I mis 
> something ?

Buffer size doesn't matter much. ReadLine() can still work properly without
getting stuck when the source's line (with EOL) has a length which is larger
than the buffer size. e.g.

  set fs = createobject("scripting.filesystemobject")
  set f = fs.createtextfile("temp.txt", true)
  f.writeline string(10000, "A")
  f.close
  set ws = createobject("wscript.shell")
  set xc = ws.exec("cmd.exe /c type temp.txt")
  wscript.echo xc.stdout.readline

If AtEndOfLine()/ReadLine()/SkipLine() is stuck, it means that either the
source's data has no EOL, or has no data at all.

[toc] | [prev] | [next] | [standalone]

#12292

JJ,

> If you need to parse the counter, you can use counted read
> one character at a time until CR is recevied, then parse the
> collected counter characters.

Yes, that could work as long as I have a single stream.  Than again, in that 
case I could just use blocking reads of just 10 chars or so (so the output 
of it to screen updates regulary) and be done with it.

The whole problem is that I have /two/ streams I need to read.  With, in 
this case, the stdout not outputting in the same frequency as stderr.

IOW: I can wait for and than read exactly one counter-outputs worth of 
characters, but than what?   There is nothing I can do with the other stream 
that won't block if there is no(t enough) output available yet.   And that 
ofcourse brings us back to square one.

> Buffer size doesn't matter much. ReadLine() can still work
> properly without getting stuck when the source's line (with
> EOL) has a length which is larger than the buffer size. e.g.

Argh.  Yes, ofcourse.   Part of it can already be loaded into the string, 
freeing up the stream buffer to receive more of the line.

Regards,
Rudy Wieser

[toc] | [prev] | [next] | [standalone]

#12293

On Sun, 21 Jun 2020 08:53:00 +0200, R.Wieser wrote:
> 
> The whole problem is that I have /two/ streams I need to read.  With, in 
> this case, the stdout not outputting in the same frequency as stderr.
> 
> IOW: I can wait for and than read exactly one counter-outputs worth of 
> characters, but than what?   There is nothing I can do with the other stream 
> that won't block if there is no(t enough) output available yet.   And that 
> ofcourse brings us back to square one.

I've been in that shoes before. i.e. two character-typed streams need to be
read, but which one is emitted first and their data length, are unknown and
can not be detected or predicted. Reading any of the stream may lead to a
wrong choice and resulting a deadlock. My conclusion was that, it's an
impossible job for TextStream. I ended up creating a custom native program
to handle the program outputs.

[toc] | [prev] | [next] | [standalone]

#12294

JJ,

> My conclusion was that, it's an impossible job
> for TextStream.

To be honest, I could not care less what method "exec" would use to buffer 
the data. :-)   As long as it allows me do stupidly basic things like a "is 
the buffer empty", or the even better "how much data is available for 
reading" check.   And perhaps a "read as much as you can, but return 
immediatily" method.

>I ended up creating a custom native program
> to handle the program outputs.

Yeah, I was already thinking of that too*, but first wanted to make sure I 
had not overlooked some dumb thing.

*and wrap it into an ActiveX component.

Thanks for the responses.

Regards,
Rudy Wieser

[toc] | [prev] | [standalone]

Back to top | Article view | microsoft.public.scripting.vbscript

csiph-web