Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.java.programmer > #8554 > unrolled thread

DataInputStream

Back to article view | Back to comp.lang.java.programmer

Contents

  DataInputStream bob <bob@coolgroups.com> - 2011-10-04 22:34 -0700
    Re: DataInputStream Robert Klemme <shortcutter@googlemail.com> - 2011-10-05 08:02 +0200
      Re: DataInputStream bob <bob@coolgroups.com> - 2011-10-04 23:17 -0700
        Re: DataInputStream Robert Klemme <shortcutter@googlemail.com> - 2011-10-05 00:07 -0700
          Re: DataInputStream bob <bob@coolgroups.com> - 2011-10-05 07:17 -0700
            Re: DataInputStream markspace <-@.> - 2011-10-05 08:27 -0700
            Re: DataInputStream Patricia Shanahan <pats@acm.org> - 2011-10-05 08:38 -0700
            Re: DataInputStream Robert Klemme <shortcutter@googlemail.com> - 2011-10-05 08:58 -0700
              Re: DataInputStream Lew <lewbloch@gmail.com> - 2011-10-05 14:52 -0700
                Re: DataInputStream Patricia Shanahan <pats@acm.org> - 2011-10-05 15:15 -0700
                Re: DataInputStream Robert Klemme <shortcutter@googlemail.com> - 2011-10-06 00:11 -0700
                  Re: DataInputStream Lew <lewbloch@gmail.com> - 2011-10-06 07:08 -0700
                    Re: DataInputStream Patricia Shanahan <pats@acm.org> - 2011-10-06 07:49 -0700
            Re: DataInputStream Eric Sosman <esosman@ieee-dot-org.invalid> - 2011-10-05 21:16 -0400
            Re: DataInputStream Arne Vajhøj <arne@vajhoej.dk> - 2011-11-06 17:48 -0500
        Re: DataInputStream Andreas Leitgeb <avl@gamma.logic.tuwien.ac.at> - 2011-10-05 08:12 +0000
          Re: DataInputStream bob <bob@coolgroups.com> - 2011-10-05 07:12 -0700
    Re: DataInputStream Roedy Green <see_website@mindprod.com.invalid> - 2011-10-06 02:48 -0700

#8554 — DataInputStream

I want to use DataInputStream to read into a float[].  Anyone know how
to do this?

[toc] | [next] | [standalone]

#8555

On 10/05/2011 07:34 AM, bob wrote:
> I want to use DataInputStream to read into a float[].  Anyone know how
> to do this?

Yes.

What did _you_ try?

Cheers

	robert

[toc] | [prev] | [next] | [standalone]

#8556

	float[] getVertices3(String filename) {

		try {
			AssetManager am = this.getResources().getAssets();
			InputStream is = am.open(filename);
			DataInputStream dis = new DataInputStream(is);
			int numfloats=dis.readInt();

			float[] floatArray = new float[numfloats];

			for (int ctr = 0; ctr < floatArray.length; ctr++) {
				floatArray[ctr] = dis.readFloat();
			}
			return floatArray;

		} catch (IOException e) {
			e.printStackTrace();
			return null;
		}

	}


On Oct 5, 1:02 am, Robert Klemme <shortcut...@googlemail.com> wrote:
> On 10/05/2011 07:34 AM, bob wrote:
>
> > I want to use DataInputStream to read into a float[].  Anyone know how
> > to do this?
>
> Yes.
>
> What did _you_ try?
>
> Cheers
>
>         robert

[toc] | [prev] | [next] | [standalone]

#8557

On Oct 5, 8:17 am, bob <b...@coolgroups.com> wrote:
>         float[] getVertices3(String filename) {
>
>                 try {
>                         AssetManager am = this.getResources().getAssets();
>                         InputStream is = am.open(filename);
>                         DataInputStream dis = new DataInputStream(is);
>                         int numfloats=dis.readInt();
>
>                         float[] floatArray = new float[numfloats];
>
>                         for (int ctr = 0; ctr < floatArray.length; ctr++) {
>                                 floatArray[ctr] = dis.readFloat();
>                         }
>                         return floatArray;
>
>                 } catch (IOException e) {
>                         e.printStackTrace();
>                         return null;
>                 }
>
>         }

That's OK.  Now, what's the issue?

Cheers

robert

[toc] | [prev] | [next] | [standalone]

#8570

The issue is that it's too slow.

What I'm hoping for is something like this:

			byte[] b = new byte[numfloats*4];
			dis.read(b, 0, numfloats*4);
			float[] f = (float[]) b;
			return f;

I don't know why, but it won't let me do the cast.  Any ideas?




On Oct 5, 2:07 am, Robert Klemme <shortcut...@googlemail.com> wrote:
> On Oct 5, 8:17 am, bob <b...@coolgroups.com> wrote:
>
>
>
>
>
>
>
>
>
> >         float[] getVertices3(String filename) {
>
> >                 try {
> >                         AssetManager am = this.getResources().getAssets();
> >                         InputStream is = am.open(filename);
> >                         DataInputStream dis = new DataInputStream(is);
> >                         int numfloats=dis.readInt();
>
> >                         float[] floatArray = new float[numfloats];
>
> >                         for (int ctr = 0; ctr < floatArray.length; ctr++) {
> >                                 floatArray[ctr] = dis.readFloat();
> >                         }
> >                         return floatArray;
>
> >                 } catch (IOException e) {
> >                         e.printStackTrace();
> >                         return null;
> >                 }
>
> >         }
>
> That's OK.  Now, what's the issue?
>
> Cheers
>
> robert

[toc] | [prev] | [next] | [standalone]

#8571

On 10/5/2011 7:17 AM, bob wrote:
> The issue is that it's too slow.
>
> What I'm hoping for is something like this:
>
> 			byte[] b = new byte[numfloats*4];
> 			dis.read(b, 0, numfloats*4);
> 			float[] f = (float[]) b;
> 			return f;
>
> I don't know why, but it won't let me do the cast.  Any ideas?


<http://download.oracle.com/javase/6/docs/api/java/nio/FloatBuffer.html#get%28float[],%20int,%20int%29>

<http://download.oracle.com/javase/6/docs/api/java/nio/ByteBuffer.html#asFloatBuffer%28%29>

[toc] | [prev] | [next] | [standalone]

#8572

On 10/5/2011 7:17 AM, bob wrote:
> The issue is that it's too slow.
>
> What I'm hoping for is something like this:
>
> 			byte[] b = new byte[numfloats*4];
> 			dis.read(b, 0, numfloats*4);
> 			float[] f = (float[]) b;
> 			return f;
>
> I don't know why, but it won't let me do the cast.  Any ideas?

Perhaps because you used a Java compiler, rather than a C compiler.

In Java, there are no guarantees about how arrays are arranged in
memory, so a cast from byte[] to float[] would not have defined results.

If it remains too slow even with buffering, you may need to consider
alternative such as matlab, Fortran, or C. Java tends not to be the
fastest language for bulk operations on floating point data.

Patricia

[toc] | [prev] | [next] | [standalone]

#8573

On Oct 5, 4:17 pm, bob <b...@coolgroups.com> wrote:
> The issue is that it's too slow.

Ah, now we're getting closer to the point.  I'd first test whether the
slowness is caused by the underlying stream or the reading procedure.
If it's the stream (e.g. because you read unbuffered from a socket)
then you might want to add buffering or you need a faster NIC.  If
it's in the reading then look at Mark's suggestion.

> What I'm hoping for is something like this:
>
>                         byte[] b = new byte[numfloats*4];
>                         dis.read(b, 0, numfloats*4);
>                         float[] f = (float[]) b;
>                         return f;
>
> I don't know why, but it won't let me do the cast.  Any ideas?

See Patricia's reply.  Java works fundamentally different from C or C+
+.  For example, there are no pointers into memory.  I seriously
suggest you make yourself familiar with the language and the JVM.

Kind regards

robert

[toc] | [prev] | [next] | [standalone]

#8581

Robert Klemme wrote:
> bob wrote:
>> The issue is that it's too slow.

How slow is that, and how fast should it be?

How much faster could it be, and what makes you think that's possible?
 
> Ah, now we're getting closer to the point.  I'd first test whether the
> slowness is caused by the underlying stream or the reading procedure.
> If it's the stream (e.g. because you read unbuffered from a socket)
> then you might want to add buffering or you need a faster NIC.  If
> it's in the reading then look at Mark's suggestion.

The OS could also be a factor.  So could other loads on the system, particularly the I/O subsystem.  Given that we don't know what speed is "too slow" and what speed is acceptable, nor the system configuration, nor the load profile, nor anything else, there's nothing we can say to affect the "too slow" evaluation.

>> What I'm hoping for is something like this:
>>
>>                         byte[] b = new byte[numfloats*4];

Did you indent enough there?

>>                         dis.read(b, 0, numfloats*4);
>>                         float[] f = (float[]) b;
>>                         return f;
>>
>> I don't know why, but it won't let me do the cast.  Any ideas?
> 
> See Patricia's reply.  Java works fundamentally different from C or C+
> +.  For example, there are no pointers into memory.  I seriously

Well, there are, actually.  In the case of the OP's code, they're 'b' and 'f'.  However, the pointers in Java are rigidly typed, unlike those in C, so you cannot cast a pointer to 'byte[]' into a pointer to 'float[]'.

> suggest you make yourself familiar with the language and the JVM.

As another respondent suggested, NIO lets you alias typed 'ByteBuffer' instances as other types of 'Buffer'.

But this defeats the purpose of 'DataInputStream', which handles those mechanics for you under the hood, so why in the world reinvent the wheel?  Just read your array of floats from the DIS.  

Have you studied the Javadocs for the type?
<http://download.oracle.com/javase/7/docs/api/java/io/DataInputStream.html>

There are links from there to relevant ancillary documentation.

-- 
Lew

[toc] | [prev] | [next] | [standalone]

#8582

On 10/5/2011 2:52 PM, Lew wrote:
...
> As another respondent suggested, NIO lets you alias typed
> 'ByteBuffer' instances as other types of 'Buffer'.
>
> But this defeats the purpose of 'DataInputStream', which handles
> those mechanics for you under the hood, so why in the world reinvent
> the wheel?  Just read your array of floats from the DIS.

I don't know whether this is one of those cases, but there are
situations in which the main limitation on data read speed is data
copying. There are two things that can reduce copy cost:

1. Do as few copies as possible.

2. When a copy is necessary, do it in bulk rather than element-by-element.

I think the NIO buffer aliasing may be a way of eliminating some
element-by-element copying.

Sometimes, in trade-offs between performance and code simplicity,
performance has to win.

Patricia

[toc] | [prev] | [next] | [standalone]

#8591

On Oct 5, 11:52 pm, Lew <lewbl...@gmail.com> wrote:
> Robert Klemme wrote:
> > bob wrote:
> >> The issue is that it's too slow.
>
> How slow is that, and how fast should it be?
>
> How much faster could it be, and what makes you think that's possible?
>
> > Ah, now we're getting closer to the point.  I'd first test whether the
> > slowness is caused by the underlying stream or the reading procedure.
> > If it's the stream (e.g. because you read unbuffered from a socket)
> > then you might want to add buffering or you need a faster NIC.  If
> > it's in the reading then look at Mark's suggestion.
>
> The OS could also be a factor.  So could other loads on the system, particularly the I/O subsystem.  Given that we don't know what speed is "too slow" and what speed is acceptable, nor the system configuration, nor the load profile, nor anything else, there's nothing we can say to affect the "too slow" evaluation.

Absolutely.  I just wanted to partition the problem space in "float
reading and parsing" and "general IO" because that information is
needed to decide whether the approach to reading floats is wrong
because it's too slow.  OS then falls into "general IO".

> >> What I'm hoping for is something like this:
>
> >>                         byte[] b = new byte[numfloats*4];
>
> Did you indent enough there?
>
> >>                         dis.read(b, 0, numfloats*4);
> >>                         float[] f = (float[]) b;
> >>                         return f;
>
> >> I don't know why, but it won't let me do the cast.  Any ideas?
>
> > See Patricia's reply.  Java works fundamentally different from C or C+
> > +.  For example, there are no pointers into memory.  I seriously
>
> Well, there are, actually.

I can see why you say that (because eventually every reference of any
kind points to a place in memory).  But for the sake of this
discussion and the general concept of Java I'd rather stick with "no
pointers into memory in Java" because that makes it crystal clear that
there is not that free access to memory cells as in C/C++.  Actually
objects can even move in memory so an unchanged reference can point to
different locations throughout its lifetime, while an unchanged C
pointer always references the same memory addresses.

>  In the case of the OP's code, they're 'b' and 'f'.  However, the pointers in Java are rigidly typed, unlike those in C, so you cannot cast a pointer to 'byte[]' into a pointer to 'float[]'.

That's why I prefer the term "object reference" which is also what the
JLS uses (even though there are some occurrences of "pointer" and we
have "NullPointerException").

Kind regards

robert

[toc] | [prev] | [next] | [standalone]

#8595

Robert Klemme wrote:
> Lew wrote:
>> Robert Klemme wrote:
>>> bob wrote:
>>>> The issue is that it's too slow.
>>
>> How slow is that, and how fast should it be?
>>
>> How much faster could it be, and what makes you think that's possible?
>>
>>> Ah, now we're getting closer to the point.  I'd first test whether the
>>> slowness is caused by the underlying stream or the reading procedure.
>>> If it's the stream (e.g. because you read unbuffered from a socket)
>>> then you might want to add buffering or you need a faster NIC.  If
>>> it's in the reading then look at Mark's suggestion.
>>
>> The OS could also be a factor.  So could other loads on the system, particularly the I/O subsystem.  Given that we don't know what speed is "too slow" and what speed is acceptable, nor the system configuration, nor the load profile, nor anything else, there's nothing we can say to affect the "too slow" evaluation.
> 
> Absolutely.  I just wanted to partition the problem space in "float
> reading and parsing" and "general IO" because that information is
> needed to decide whether the approach to reading floats is wrong
> because it's too slow.  OS then falls into "general IO".

If "general IO" takes 100 ms and the specifics of the float conversion take 100 μs (to pick numbers out of my, er, hat), then time spent on the latter should wait until the former is properly handled.  You might shave a whopping 50 μs out of your 100100 μs total overhad if you double the speed of the wrong side.  You'll still have roughly 99.95% of your overhead.

It's not enough to solve the problem; you have to solve the right problem.

That said, the partition is useful.  It helps you organize your search for the high-value targets.

>>>> What I'm hoping for is something like this:
>>
>>>>                         byte[] b = new byte[numfloats*4];
>>
>> Did you indent enough there?
>>
>>>>                         dis.read(b, 0, numfloats*4);
>>>>                         float[] f = (float[]) b;
>>>>                         return f;
>>
>>>> I don't know why, but it won't let me do the cast.  Any ideas?
>>
>>> See Patricia's reply.  Java works fundamentally different from C or C+
>>> +.  For example, there are no pointers into memory.  I seriously
>>
>> Well, there are, actually.
> 
> I can see why you say that (because eventually every reference of any
> kind points to a place in memory).  But for the sake of this
> discussion and the general concept of Java I'd rather stick with "no
> pointers into memory in Java" because that makes it crystal clear that
> there is not that free access to memory cells as in C/C++.  Actually

I'd rather stick with the truth to make it crystal clear what's true.

Imprecise analysis leads to incorrect solutions.

The truth is that Java has pointers.  The nuance is that these pointers do not behave like those in C, but then C didn't invent pointers to begin with.

> objects can even move in memory so an unchanged reference can point to
> different locations throughout its lifetime, while an unchanged C
> pointer always references the same memory addresses.

So say that, rather than the imprecise and incorrect canard that "Java doesn't have pointers".

>>  In the case of the OP's code, they're 'b' and 'f'.  However, the pointers in Java are rigidly typed, unlike those in C, so you cannot cast a pointer to 'byte[]' into a pointer to 'float[]'.
> 
> That's why I prefer the term "object reference" which is also what the
> JLS uses (even though there are some occurrences of "pointer" and we
> have "NullPointerException").

The JLS flat out defines the terms "pointer" and "reference" as equivalent.

The word "object" in "object reference" is redundant, and not correct in the case when the reference points (!) to 'null'.

But do feel free to use the term "reference", which in Java's nomenclature is exactly equivalent to "pointer".

The key feature of a pointer is that it points to something.  The ability to alias pointer types (e.g., 'byte' array to 'float' array), perform pointer arithmetic or to point to invalid memory are not essential to the definition of "pointer".

Let's not lie to people in an attempt to dumb the information down.  Let's instead tell the truth and require programmers to be smart.  There's far too much dumb programming in the wild already.

-- 
Lew

[toc] | [prev] | [next] | [standalone]

#8597

On 10/6/2011 7:08 AM, Lew wrote:
...
> But do feel free to use the term "reference", which in Java's
> nomenclature is exactly equivalent to "pointer".
>
> The key feature of a pointer is that it points to something.  The
> ability to alias pointer types (e.g., 'byte' array to 'float' array),
> perform pointer arithmetic or to point to invalid memory are not
> essential to the definition of "pointer".
...

<pedant>
According to the JLS, "The reference values (often just references) are
pointers to these objects, and a special null reference, which refers to
no object."

That is, a pointer always points to something. A reference may be null,
in which case it does not point to anything and therefore is not a pointer.
<\pedant>

Patricia

[toc] | [prev] | [next] | [standalone]

#8587

On 10/5/2011 10:17 AM, bob wrote:
> The issue is that it's too slow.
>
> What I'm hoping for is something like this:
>
> 			byte[] b = new byte[numfloats*4];
> 			dis.read(b, 0, numfloats*4);
> 			float[] f = (float[]) b;
> 			return f;
>
> I don't know why, but it won't let me do the cast.  Any ideas?

     "Why" is because an array of byte is not an array of float.
Objects in Java (an array is an object) are not just blobs of
memory; they have more identity, more selfhood.

     The wider problem you seem to be struggling with is that
reading these numbers is "slow as molasses" (as you so precisely
put it in a related thread).  Several people have suggested that
you (a) refine your notions of slowness and (b) take various
steps to determine what contributes to it.  As far as I can see
you've done neither: You're simply assuming (on no basis that's
been revealed) that the process of assembling bytes into floats is
the fount of treacle.  That's conceivable, just, but strikes me as
highly unlikely.  Go, and sin no more: Make some measurements of
where the time goes, and attack the measured hot spots instead of
your imagined phantoms.

-- 
Eric Sosman
esosman@ieee-dot-org.invalid

[toc] | [prev] | [next] | [standalone]

#9691

On 10/5/2011 10:17 AM, bob wrote:
> On Oct 5, 2:07 am, Robert Klemme<shortcut...@googlemail.com>  wrote:
>> On Oct 5, 8:17 am, bob<b...@coolgroups.com>  wrote:
>>>          float[] getVertices3(String filename) {
>>
>>>                  try {
>>>                          AssetManager am = this.getResources().getAssets();
>>>                          InputStream is = am.open(filename);
>>>                          DataInputStream dis = new DataInputStream(is);
>>>                          int numfloats=dis.readInt();
>>
>>>                          float[] floatArray = new float[numfloats];
>>
>>>                          for (int ctr = 0; ctr<  floatArray.length; ctr++) {
>>>                                  floatArray[ctr] = dis.readFloat();
>>>                          }
>>>                          return floatArray;
>>
>>>                  } catch (IOException e) {
>>>                          e.printStackTrace();
>>>                          return null;
>>>                  }
>>
>>>          }
>>
>> That's OK.  Now, what's the issue?

> The issue is that it's too slow.

Rather unlikely that readFloat is the culprit.

Try replace:

InputStream is = am.open(filename);
DataInputStream dis = new DataInputStream(is);

with:

InputStream is = new BufferedInputStream(am.open(filename), 65536);
DataInputStream dis = new DataInputStream(is);

Arne

[toc] | [prev] | [next] | [standalone]

#8559

bob <bob@coolgroups.com> wrote:
> 	float[] getVertices3(String filename) {
> 		try {
> 			AssetManager am = this.getResources().getAssets();
> 			InputStream is = am.open(filename);
> 			DataInputStream dis = new DataInputStream(is);
> 			int numfloats=dis.readInt();
> 			float[] floatArray = new float[numfloats];
> 			for (int ctr = 0; ctr < floatArray.length; ctr++) {
> 				floatArray[ctr] = dis.readFloat();
> 			}
> 			return floatArray;

Are you aware, that the DataInputStream expects a particular *binary* 
format of data coming from the stream?

If the data stored in the file that you use still contains the same
textual representation as for your previous questions, then
DataInputStream won't be able to read it properly.

You might create a separate converter utility, that reads in your
textfile (with StreamTokenizer), then creates a binary file (writing
out size and the individual floats through a DataOutputStream.
That may seem roundabout, but is just right, if you *casually* need
to hand-edit the text-file, but want speed-improved reading into your
main application.

Another option could be reading the float[] as a single object from
an ObjectInputStream. Again. the data for that must have been created
before by writing such an array to an ObjectOutputStream.

[toc] | [prev] | [next] | [standalone]

#8569

Yes, I'm aware.  I already had to change the endian-ness when the data
was written.  Thanks.



On Oct 5, 3:12 am, Andreas Leitgeb <a...@gamma.logic.tuwien.ac.at>
wrote:
> bob <b...@coolgroups.com> wrote:
> >    float[] getVertices3(String filename) {
> >            try {
> >                    AssetManager am = this.getResources().getAssets();
> >                    InputStream is = am.open(filename);
> >                    DataInputStream dis = new DataInputStream(is);
> >                    int numfloats=dis.readInt();
> >                    float[] floatArray = new float[numfloats];
> >                    for (int ctr = 0; ctr < floatArray.length; ctr++) {
> >                            floatArray[ctr] = dis.readFloat();
> >                    }
> >                    return floatArray;
>
> Are you aware, that the DataInputStream expects a particular *binary*
> format of data coming from the stream?
>
> If the data stored in the file that you use still contains the same
> textual representation as for your previous questions, then
> DataInputStream won't be able to read it properly.
>
> You might create a separate converter utility, that reads in your
> textfile (with StreamTokenizer), then creates a binary file (writing
> out size and the individual floats through a DataOutputStream.
> That may seem roundabout, but is just right, if you *casually* need
> to hand-edit the text-file, but want speed-improved reading into your
> main application.
>
> Another option could be reading the float[] as a single object from
> an ObjectInputStream. Again. the data for that must have been created
> before by writing such an array to an ObjectOutputStream.

[toc] | [prev] | [next] | [standalone]

#8593

On Tue, 4 Oct 2011 22:34:56 -0700 (PDT), bob <bob@coolgroups.com>
wrote, quoted or indirectly quoted someone who said :

>I want to use DataInputStream to read into a float[].  Anyone know how
>to do this?

In Java, with DataInputStream you have to think in terms of reading
primitives, not records as you would in C++ or COBOL.

You would read a count followed by n floats, one field at a time. Ask
http://mindprod.com/jgloss/fileio.html to generate you the code.

You can logically read and write entire objects including arrays with
dependent contents using serialization.  See
http://mindprod.com/jgloss/serialization.html
again http://mindprod.com/jgloss/fileio.html will generate you sample
code.

Part of the reason is Java as WORA. The I/O has to work whether the
machine is internally in big or little ended.  the run time has to
convert on a field by field basis.  Java can't treat objects/records
as just a blob of bytes the way earlier machine-dependent languages
can.

It is a lot easier to let java determine the exact binary layout of
files rather that trying to read some bizarre format created
elsewhere.  Then you have to read byte by byte with an InputStream.


-- 
Roedy Green Canadian Mind Products
http://mindprod.com
It should not be considered an error when the user starts something
already started or stops something already stopped. This applies
to browsers, services, editors... It is inexcusable to 
punish the user by requiring some elaborate sequence to atone,
e.g. open the task editor, find and kill some processes.

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.java.programmer

csiph-web