Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.java.programmer > #38952 > unrolled thread

readLine() and newline problem

Back to article view | Back to comp.lang.java.programmer

Contents

  readLine() and newline problem mike <mikaelpetterson@hotmail.com> - 2019-05-09 23:16 -0700
    Re: readLine() and newline problem Daniele Futtorovic <da.futt.news@laposte-dot-net.invalid> - 2019-05-10 09:02 +0200
      Re: readLine() and newline problem mike <mikaelpetterson@hotmail.com> - 2019-05-10 01:29 -0700
        Re: readLine() and newline problem Martin Gregorie <martin@mydomain.invalid> - 2019-05-10 10:22 +0000
        Re: readLine() and newline problem Daniele Futtorovic <da.futt.news@laposte-dot-net.invalid> - 2019-05-10 18:50 +0200

#38952 — readLine() and newline problem

Hi,

We have the following code for reading a NETCONF "hello" message from a NETCONF Device.

public class MessageHandler  {

        public StringBuilder readLines(InputStream is) {
           final StringBuilder buffer = new StringBuilder();
           final String delimiter = "]]>]]>";
           String tmp;
           
           try (BufferedReader reader = new BufferedReader(new InputStreamReader(is))) {
               do {

                   tmp = reader.readLine();

                   if (tmp != null) {
                       buffer.append(tmp);
                   }
               } while (tmp != null && !tmp.endsWith(delimiter));

           } catch (IOException e) {
               LOGGER.info("Failed to read <rpc-reply> message ",
                       e);
           }
           
           return buffer;
           
       }
        
       


}

We get the following string from the device ( just an example):

<?xml version="1.0" encoding="UTF-8"?>
    <hello xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
      <capabilities>
        <capability>
          urn:ietf:params:netconf:base:1.1
        </capability>
        <capability>
          urn:ietf:params:ns:netconf:capability:startup:1.0
        </capability>
      </capabilities>
      <session-id>4</session-id>
    </hello>
    ]]>]]>

The problem here is that some devices add a newline after "]]>]]>" which is the message separator. And of course some devices don't since it is not required in the RFC6242.

So when there is no newline from device then the above method will hang. Then I thought it would be easy to create a unit test that recreates the problem. One with a hello message with message separator and newline and one with only message separator. But problem is that I get no exception in the test without the newline. Any ideas what I am missing or how I can re-create the issue?

br,

//mike

public class MessageHandlerTest {

    private MessageHandler messageHandler;

    @BeforeClass
    public void setup() {
        messageHandler = new MessageHandler();
    }

    @Test
    public void testHelloServerWithOutNewLine() throws Exception {
        InputStream is = getHelloWithoutNewline();
        StringBuilder actual = messageHandler.readLines(is);
        Assert.assertTrue(actual.length() > 0);

    }

    @Test
    public void testHelloServerWithNewline() throws Exception {
        InputStream is = getHelloWithNewline();
        StringBuilder actual = messageHandler.readLines(is);
        Assert.assertTrue(actual.length() > 0);

    }

    
    private InputStream getHelloWithoutNewline() {
        final String HELLO_REPLY = "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n" + 
                "   <hello xmlns=\"urn:ietf:params:xml:ns:netconf:base:1.0\">" + 
                "      <capabilities>" + 
                "        <capability>" + 
                "          urn:ietf:params:netconf:base:1.1" + 
                "        </capability>" + 
                "        <capability>" + 
                "          urn:ietf:params:ns:netconf:capability:startup:1.0" + 
                "        </capability>" + 
                "      </capabilities>" + 
                "      <session-id>4</session-id>" + 
                "    </hello>" + 
                "    ]]>]]>";
        
        return new ByteArrayInputStream(HELLO_REPLY.getBytes());

    }

    private InputStream getHelloWithNewline() {
        final String HELLO_REPLY = "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n" + 
                "   <hello xmlns=\"urn:ietf:params:xml:ns:netconf:base:1.0\">\n" + 
                "      <capabilities>\n" + 
                "        <capability>\n" + 
                "          urn:ietf:params:netconf:base:1.1\n" + 
                "        </capability>\n" + 
                "        <capability>\n" + 
                "          urn:ietf:params:ns:netconf:capability:startup:1.0\n" + 
                "        </capability>\n" + 
                "      </capabilities>\n" + 
                "      <session-id>4</session-id>\n" + 
                "    </hello>\n" + 
                "    ]]>]]>\n";
        
        return new ByteArrayInputStream(HELLO_REPLY.getBytes());

    }

}

[toc] | [next] | [standalone]

#38953

On 2019-05-10 08:16, mike wrote:
> The problem here is that some devices add a newline after "]]>]]>" which is the message separator. And of course some devices don't since it is not required in the RFC6242.
> 
> So when there is no newline from device then the above method will hang. Then I thought it would be easy to create a unit test that recreates the problem. One with a hello message with message separator and newline and one with only message separator. But problem is that I get no exception in the test without the newline. Any ideas what I am missing or how I can re-create the issue?

In your test, an EOF takes the place of the otherwise expected EOL and
makes it work.

As for the problem overall, in my view you definitely shouldn't be using
readLine() here. You should be reading with read(char[]) and keep doing
so until you encounter the separator.

-- 
DF.

[toc] | [prev] | [next] | [standalone]

#38954

Den fredag 10 maj 2019 kl. 09:02:49 UTC+2 skrev Daniele Futtorovic:
> On 2019-05-10 08:16, mike wrote:
> > The problem here is that some devices add a newline after "]]>]]>" which is the message separator. And of course some devices don't since it is not required in the RFC6242.
> > 
> > So when there is no newline from device then the above method will hang. Then I thought it would be easy to create a unit test that recreates the problem. One with a hello message with message separator and newline and one with only message separator. But problem is that I get no exception in the test without the newline. Any ideas what I am missing or how I can re-create the issue?
> 
> In your test, an EOF takes the place of the otherwise expected EOL and
> makes it work.
> 
> As for the problem overall, in my view you definitely shouldn't be using
> readLine() here. You should be reading with read(char[]) and keep doing
> so until you encounter the separator.
> 
> -- 
> DF.

So how can I update test to simulate my error?

Yes I am also thinking of using read(char []). I think it is more appropriate. The only thing that will be a bit tricky is to find my separator char.I have to check if we have ] then another ] so there will be many ifs.

[toc] | [prev] | [next] | [standalone]

#38955

On Fri, 10 May 2019 01:29:43 -0700, mike wrote:

> Den fredag 10 maj 2019 kl. 09:02:49 UTC+2 skrev Daniele Futtorovic:
>> On 2019-05-10 08:16, mike wrote:
>> > The problem here is that some devices add a newline after "]]>]]>"
>> > which is the message separator. And of course some devices don't
>> > since it is not required in the RFC6242.
>> > 
>> > So when there is no newline from device then the above method will
>> > hang. Then I thought it would be easy to create a unit test that
>> > recreates the problem. One with a hello message with message
>> > separator and newline and one with only message separator. But
>> > problem is that I get no exception in the test without the newline.
>> > Any ideas what I am missing or how I can re-create the issue?
>> 
>> In your test, an EOF takes the place of the otherwise expected EOL and
>> makes it work.
>> 
>> As for the problem overall, in my view you definitely shouldn't be
>> using readLine() here. You should be reading with read(char[]) and keep
>> doing so until you encounter the separator.
>> 
>> --
>> DF.
> 
> So how can I update test to simulate my error?
> 
> Yes I am also thinking of using read(char []). I think it is more
> appropriate. The only thing that will be a bit tricky is to find my
> separator char.I have to check if we have ] then another ] so there will
> be many ifs.

Two suggestions:

1) Continue to use readLine() but use Scanner.findInLine() to recognise 
the end pattern and DON'T include a newline in the end pattern. 

2) Read everything into a String and then use a Scanner or 
String.indexOf(String.str) to find the end pattern and truncate the 
string at that point. 

Which one is better depends on how much unwanted stuff follows the 
terminator and how much time would be wasted reading it in and discarding 
it.  


-- 
Martin    | martin at
Gregorie  | gregorie dot org

[toc] | [prev] | [next] | [standalone]

#38956

On 2019-05-10 10:29, mike wrote:
> So how can I update test to simulate my error?

That's a bit trickier, but the code below should give you an idea.

> Yes I am also thinking of using read(char []). I think it is more appropriate. The only thing that will be a bit tricky is to find my separator char.I have to check if we have ] then another ] so there will be many ifs.

Save yourself trouble and use a java.util.Scanner. See below.

Code:
````
import org.junit.Test;

import java.io.*;
import java.util.Objects;
import java.util.Scanner;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;

public class NoEol {

    static final String DATA =
        "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n"
        + "    <hello xmlns=\"urn:ietf:params:xml:ns:netconf:base:1.0\">\n"
        + "      <capabilities>\n"
        + "        <capability>\n"
        + "          urn:ietf:params:netconf:base:1.1\n"
        + "        </capability>\n"
        + "        <capability>\n"
        + "          urn:ietf:params:ns:netconf:capability:startup:1.0\n"
        + "        </capability>\n"
        + "      </capabilities>\n"
        + "      <session-id>4</session-id>\n"
        + "    </hello>\n"
        + "    ]]>]]>"; //NOTE no newline at the end

    @Test
    public void testBufferedReaderWithNewLine(){
        try(Supplier s = Supplier.newInstance(DATA + "\n")){
            globWithBufferedReader( s.input(), "]]>]]>");
        }
        catch ( IOException e ) {
            e.printStackTrace();
        }
    }

    @Test
    public void testBufferedReaderWithoutNewLine(){
        try(Supplier s = Supplier.newInstance(DATA)){
            globWithBufferedReader( s.input(), "]]>]]>");
        }
        catch ( IOException e ) {
            e.printStackTrace();
        }
    }

    @Test
    public void testScannerWithNewLine(){
        try(Supplier s = Supplier.newInstance(DATA + "\n")){
            globWithScanner( s.input(), "]]>]]>");
        }
    }

    @Test
    public void testScannerWithoutNewLine(){
        try(Supplier s = Supplier.newInstance(DATA)){
            globWithScanner( s.input(), "]]>]]>");
        }
    }

    static void globWithBufferedReader( Reader r, String sep) throws
IOException {
        BufferedReader br = new BufferedReader(r);
        StringBuilder sb = new StringBuilder();

        for(String line; null != (line = br.readLine()); ){
            sb.append(line);

            if( line.contains( sep ) ){
                break;
            }
        }

        System.out.println("Read finished: " + sb);
    }

    static void globWithScanner(Reader r, String sep){
        Scanner sc = new Scanner(r);
        sc.useDelimiter( sep );

        System.out.println("Read finished: " + sc.next());
    }

    private static final class Supplier implements Runnable, AutoCloseable {
        static final ExecutorService executor =
Executors.newCachedThreadPool();

        private final String data;

        private final PipedReader pipeIn;
        private final PipedWriter pipeOut;

        private volatile Thread thread;

        Supplier( String data )
        {
            this.data = Objects.requireNonNull( data, "data" );

            pipeOut = new PipedWriter();

            try {
                pipeIn = new PipedReader( pipeOut );
            }
            catch ( IOException x ) {
                throw new AssertionError( "Could not cerate pipe", x );
            }
        }

        public Reader input() {
            return pipeIn;
        }

        @Override
        public void close() {
            if ( thread != null ) {
                thread.interrupt();
                thread = null;
            }
        }

        @Override
        public void run() {
            thread = Thread.currentThread();

            try {
                pipeOut.write( data );

                synchronized ( this ) {
                    wait();
                }
            }
            catch ( InterruptedException e ) {
                System.err.println( "Interrupted." );
                Thread.currentThread().interrupt();
            }
            catch ( IOException e ) {
                e.printStackTrace();
            }

            System.err.println( "Supplier exiting." );
        }

        public static Supplier newInstance( String data ) {
            Supplier ret = new Supplier(data);
            executor.submit( ret );

            return ret;
        }
    }
}
````
-- 
DF.

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.java.programmer

csiph-web