Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #64410

Flushing out data from Popen buffer

Show all headers | View raw

I am working on a python script that reads data by tailing a file and then puts in a different file. The script works in a time bound manner and eventually flushes out the data from the buffer when the ENDTIME is reached. However there has been a mismatch in the source and target file in terms of size.

Following is a snippet:

    self.read_size = 2048
    self.tail_buffer = 2048

    # start the file tail
    cmd = '%s -f -o %s %s' % (self.jtailcmd, offset, self.source_journal)
    self.logger.debug('[%s] starting FILETail' % self.getName())
    try:
        self.jtail = popen2.Popen3(cmd, bufsize = self.tail_buffer)
        self.jtail.tochild.close()
        out = self.jtail.fromchild
        outfd = self.jtail.fromchild.fileno()
        flags = fcntl.fcntl(outfd, fcntl.F_GETFL)
        fcntl.fcntl(outfd, fcntl.F_SETFL, flags | os.O_NONBLOCK)
    except:
        message = '[%s] error reading file' % self.getName()
        self.logger.error(message)
        self.logger.error('[%s] %s: %s' % \
            (self.getName(), sys.exc_info()[0], sys.exc_info()[1]))
        send_alert('AE', message)
        self.sleep(60)
        self.close_tail()
        self.close_ssh()
And then eventually it flushes out the data:

        try:
            [i, o, e] = select.select([outfd], [], [], 1)
            if i:
                data = out.read(self.read_size)
            else:
                data = None
        except:
            message = '[%s] error reading file' % self.getName()
            self.logger.error(message)
            self.logger.error('[%s] %s: %s' % \
                (self.getName(), sys.exc_info()[0], sys.exc_info()[1]))
            send_alert('AE', message)
            self.close_tail()
            self.close_ssh()
            self.sleep(60)
            break
        if data:
            if self.sshcat.poll() != -1:
                self.logger.error('[%s] connection error' % self.getName())
                self.close_tail()
                self.close_ssh()
                break
            try:
                self.sshcat.tochild.writelines(data)
                self.sshcat.tochild.flush()
            except:
                message = '[%s] error writing remote file' % self.getName()
While troubleshooting, I narrowed out the problem to tail_buffer size! By reducing the tail_buffer size , the script worked fine. I cant use subprocess(Python Version is 2.4.3)

I do not want to rely on tail_buffer size. Ideally the script should be independent of it!

Is there a way to flush data from the POPEN buffer ?

Back to comp.lang.python | Previous | Next | Find similar | Unroll thread

Thread

Flushing out data from Popen buffer abc79721 <karanchawla53@gmail.com> - 2014-01-21 05:46 -0800

csiph-web