Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]


Groups > comp.lang.java.programmer > #11821 > unrolled thread

Re: Interplatform (interprocess, interlanguage) communication

Started byjebblue <n@n.nnn>
First post2012-02-07 12:11 -0600
Last post2012-02-08 00:55 -0700
Articles 20 on this page of 70 — 7 participants

Back to article view | Back to comp.lang.java.programmer

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.


Contents

  Re: Interplatform (interprocess, interlanguage) communication jebblue <n@n.nnn> - 2012-02-07 12:11 -0600
    Re: Interplatform (interprocess, interlanguage) communication BGB <cr88192@hotmail.com> - 2012-02-07 16:38 -0700
      Re: Interplatform (interprocess, interlanguage) communication Arved Sandstrom <asandstrom3minus1@eastlink.ca> - 2012-02-07 20:26 -0400
        Re: Interplatform (interprocess, interlanguage) communication BGB <cr88192@hotmail.com> - 2012-02-08 01:41 -0700
          Re: Interplatform (interprocess, interlanguage) communication Arved Sandstrom <asandstrom3minus1@eastlink.ca> - 2012-02-08 07:19 -0400
            Re: Interplatform (interprocess, interlanguage) communication BGB <cr88192@hotmail.com> - 2012-02-08 12:07 -0700
              Re: Interplatform (interprocess, interlanguage) communication Arne Vajhøj <arne@vajhoej.dk> - 2012-02-08 21:16 -0500
                Re: Interplatform (interprocess, interlanguage) communication BGB <cr88192@hotmail.com> - 2012-02-08 19:50 -0700
                  Re: Interplatform (interprocess, interlanguage) communication Arved Sandstrom <asandstrom3minus1@eastlink.ca> - 2012-02-09 06:24 -0400
                    Re: Interplatform (interprocess, interlanguage) communication BGB <cr88192@hotmail.com> - 2012-02-09 09:15 -0700
                      Re: Interplatform (interprocess, interlanguage) communication Arved Sandstrom <asandstrom3minus1@eastlink.ca> - 2012-02-09 18:58 -0400
                        Re: Interplatform (interprocess, interlanguage) communication BGB <cr88192@hotmail.com> - 2012-02-09 16:15 -0700
                        Re: Interplatform (interprocess, interlanguage) communication Arne Vajhøj <arne@vajhoej.dk> - 2012-02-09 18:50 -0500
                          Re: Interplatform (interprocess, interlanguage) communication BGB <cr88192@hotmail.com> - 2012-02-09 21:40 -0700
                            Re: Interplatform (interprocess, interlanguage) communication Arne Vajhøj <arne@vajhoej.dk> - 2012-02-11 14:47 -0500
                              Re: Interplatform (interprocess, interlanguage) communication Lew <lewbloch@gmail.com> - 2012-02-11 12:06 -0800
                                Re: Interplatform (interprocess, interlanguage) communication Arne Vajhøj <arne@vajhoej.dk> - 2012-02-11 15:18 -0500
                                  Re: Interplatform (interprocess, interlanguage) communication BGB <cr88192@hotmail.com> - 2012-02-11 23:03 -0700
                                    Re: Interplatform (interprocess, interlanguage) communication Arne Vajhøj <arne@vajhoej.dk> - 2012-02-12 09:27 -0500
                                      Re: Interplatform (interprocess, interlanguage) communication BGB <cr88192@hotmail.com> - 2012-02-12 13:33 -0700
                                        Re: Interplatform (interprocess, interlanguage) communication Arne Vajhøj <arne@vajhoej.dk> - 2012-02-12 15:50 -0500
                                          Re: Interplatform (interprocess, interlanguage) communication BGB <cr88192@hotmail.com> - 2012-02-12 14:34 -0700
                      Re: Interplatform (interprocess, interlanguage) communication Arne Vajhøj <arne@vajhoej.dk> - 2012-02-09 18:48 -0500
                        Re: Interplatform (interprocess, interlanguage) communication BGB <cr88192@hotmail.com> - 2012-02-09 21:46 -0700
                          Re: Interplatform (interprocess, interlanguage) communication Lew <lewbloch@gmail.com> - 2012-02-10 08:51 -0800
                            Re: Interplatform (interprocess, interlanguage) communication BGB <cr88192@hotmail.com> - 2012-02-10 10:43 -0700
                              Re: Interplatform (interprocess, interlanguage) communication Lew <lewbloch@gmail.com> - 2012-02-10 13:15 -0800
                                Re: Interplatform (interprocess, interlanguage) communication BGB <cr88192@hotmail.com> - 2012-02-10 14:50 -0700
                                  Re: Interplatform (interprocess, interlanguage) communication Lew <lewbloch@gmail.com> - 2012-02-10 14:32 -0800
                                    Re: Interplatform (interprocess, interlanguage) communication BGB <cr88192@hotmail.com> - 2012-02-10 17:10 -0700
                                      Re: Interplatform (interprocess, interlanguage) communication Arved Sandstrom <asandstrom3minus1@eastlink.ca> - 2012-02-10 22:08 -0400
                                        Re: Interplatform (interprocess, interlanguage) communication BGB <cr88192@hotmail.com> - 2012-02-11 00:49 -0700
                                          Re: Interplatform (interprocess, interlanguage) communication Arved Sandstrom <asandstrom3minus1@eastlink.ca> - 2012-02-11 14:04 -0400
                                Re: Interplatform (interprocess, interlanguage) communication Arne Vajhøj <arne@vajhoej.dk> - 2012-02-11 14:55 -0500
                              Re: Interplatform (interprocess, interlanguage) communication Arne Vajhøj <arne@vajhoej.dk> - 2012-02-11 14:52 -0500
                                Re: Interplatform (interprocess, interlanguage) communication BGB <cr88192@hotmail.com> - 2012-02-11 20:06 -0700
                                  Re: Interplatform (interprocess, interlanguage) communication Arne Vajhøj <arne@vajhoej.dk> - 2012-02-11 22:41 -0500
                                    Re: Interplatform (interprocess, interlanguage) communication BGB <cr88192@hotmail.com> - 2012-02-12 00:46 -0700
                                      Re: Interplatform (interprocess, interlanguage) communication Arne Vajhøj <arne@vajhoej.dk> - 2012-02-12 09:29 -0500
                                      Re: Interplatform (interprocess, interlanguage) communication Arne Vajhøj <arne@vajhoej.dk> - 2012-02-12 09:31 -0500
                                    Re: Interplatform (interprocess, interlanguage) communication Martin Gregorie <martin@address-in-sig.invalid> - 2012-02-12 16:02 +0000
                                      Re: Interplatform (interprocess, interlanguage) communication Arne Vajhøj <arne@vajhoej.dk> - 2012-02-12 11:16 -0500
                                        Re: Interplatform (interprocess, interlanguage) communication Martin Gregorie <martin@address-in-sig.invalid> - 2012-02-12 22:46 +0000
                                      Re: Interplatform (interprocess, interlanguage) communication BGB <cr88192@hotmail.com> - 2012-02-12 11:33 -0700
                                  Re: Interplatform (interprocess, interlanguage) communication Lew <lewbloch@gmail.com> - 2012-02-11 20:18 -0800
                                    Re: Interplatform (interprocess, interlanguage) communication BGB <cr88192@hotmail.com> - 2012-02-12 01:36 -0700
                                      Re: Interplatform (interprocess, interlanguage) communication Joshua Cranmer <Pidgeot18@verizon.invalid> - 2012-02-12 13:52 -0600
                                        Re: Interplatform (interprocess, interlanguage) communication BGB <cr88192@hotmail.com> - 2012-02-12 14:43 -0700
                          Re: Interplatform (interprocess, interlanguage) communication Arne Vajhøj <arne@vajhoej.dk> - 2012-02-11 14:49 -0500
                    Re: Interplatform (interprocess, interlanguage) communication Arne Vajhøj <arne@vajhoej.dk> - 2012-02-09 18:46 -0500
                  Re: Interplatform (interprocess, interlanguage) communication Arne Vajhøj <arne@vajhoej.dk> - 2012-02-09 18:45 -0500
          Re: Interplatform (interprocess, interlanguage) communication Lew <lewbloch@gmail.com> - 2012-02-08 14:02 -0800
            Re: Interplatform (interprocess, interlanguage) communication BGB <cr88192@hotmail.com> - 2012-02-08 18:49 -0700
              Re: Interplatform (interprocess, interlanguage) communication Arne Vajhøj <arne@vajhoej.dk> - 2012-02-08 21:14 -0500
                Re: Interplatform (interprocess, interlanguage) communication Lew <lewbloch@gmail.com> - 2012-02-08 20:07 -0800
                  Re: Interplatform (interprocess, interlanguage) communication BGB <cr88192@hotmail.com> - 2012-02-08 23:29 -0700
                    Re: Interplatform (interprocess, interlanguage) communication Lew <lewbloch@gmail.com> - 2012-02-09 09:40 -0800
                      Re: Interplatform (interprocess, interlanguage) communication BGB <cr88192@hotmail.com> - 2012-02-09 17:02 -0700
                Re: Interplatform (interprocess, interlanguage) communication BGB <cr88192@hotmail.com> - 2012-02-08 21:10 -0700
                  Re: Interplatform (interprocess, interlanguage) communication Arne Vajhøj <arne@vajhoej.dk> - 2012-02-09 18:54 -0500
                    Re: Interplatform (interprocess, interlanguage) communication BGB <cr88192@hotmail.com> - 2012-02-10 10:25 -0700
                      Re: Interplatform (interprocess, interlanguage) communication Arne Vajhøj <arne@vajhoej.dk> - 2012-02-11 14:45 -0500
                        Re: Interplatform (interprocess, interlanguage) communication Lew <lewbloch@gmail.com> - 2012-02-11 12:14 -0800
                          Re: Interplatform (interprocess, interlanguage) communication Arne Vajhøj <arne@vajhoej.dk> - 2012-02-11 15:20 -0500
                            Re: Interplatform (interprocess, interlanguage) communication BGB <cr88192@hotmail.com> - 2012-02-11 22:20 -0700
                              Re: Interplatform (interprocess, interlanguage) communication Arne Vajhøj <arne@vajhoej.dk> - 2012-02-12 09:23 -0500
                                Re: Interplatform (interprocess, interlanguage) communication BGB <cr88192@hotmail.com> - 2012-02-12 12:13 -0700
      Re: Interplatform (interprocess, interlanguage) communication Arne Vajhøj <arne@vajhoej.dk> - 2012-02-07 20:24 -0500
      Re: Interplatform (interprocess, interlanguage) communication Martin Gregorie <martin@address-in-sig.invalid> - 2012-02-08 01:31 +0000
        Re: Interplatform (interprocess, interlanguage) communication BGB <cr88192@hotmail.com> - 2012-02-08 00:55 -0700

Page 1 of 4  [1] 2 3 4  Next page →


#11821 — Re: Interplatform (interprocess, interlanguage) communication

Fromjebblue <n@n.nnn>
Date2012-02-07 12:11 -0600
SubjectRe: Interplatform (interprocess, interlanguage) communication
Message-ID<aM6dnWFo75_W9KzSnZ2dnUVZ_sqdnZ2d@giganews.com>
On Fri, 03 Feb 2012 19:52:08 +0000, Stefan Ram wrote:

> »X« below is another language than Java, for example,
>   VBA, C#, or C.
> 
>   When an X process and a Java process have to exchange information on
>   the same computer, what possibilites are there? The Java process
>   should act as a client, sending commands to the X process and also
>   wants to read answers from the X process. So, the X process is a kind
>   of server.
> 
>   My criteria are: reliability and it should not be extremely slow (say
>   exchanging a string should not take more than about 10 ms). The main
>   criterion is reliability.
> 

>   Sockets
> 
>   This is slightly less transparent than files, but has the advantage
>   that it becomes very easy to have the two processes running on
>   different computers later, if this should ever be required. Debugging
>   should be possible by a man-in-the-middle proxy that prints all
>   information it sees or by connecting to the server with a terminal.
> 

I recommend using sockets.

-- 
// This is my opinion.

[toc] | [next] | [standalone]


#11835

FromBGB <cr88192@hotmail.com>
Date2012-02-07 16:38 -0700
Message-ID<jgscnm$1td$1@news.albasani.net>
In reply to#11821
On 2/7/2012 11:11 AM, jebblue wrote:
> On Fri, 03 Feb 2012 19:52:08 +0000, Stefan Ram wrote:
>
>> »X« below is another language than Java, for example,
>>    VBA, C#, or C.
>>
>>    When an X process and a Java process have to exchange information on
>>    the same computer, what possibilites are there? The Java process
>>    should act as a client, sending commands to the X process and also
>>    wants to read answers from the X process. So, the X process is a kind
>>    of server.
>>
>>    My criteria are: reliability and it should not be extremely slow (say
>>    exchanging a string should not take more than about 10 ms). The main
>>    criterion is reliability.
>>
>
>>    Sockets
>>
>>    This is slightly less transparent than files, but has the advantage
>>    that it becomes very easy to have the two processes running on
>>    different computers later, if this should ever be required. Debugging
>>    should be possible by a man-in-the-middle proxy that prints all
>>    information it sees or by connecting to the server with a terminal.
>>
>
> I recommend using sockets.
>

in general, I agree (sockets generally make the most sense), although 
there are cases where file-based communications can make sense, although 
probably not in the form as described in the OP.


another issue (besides how to pass messages), is what sort of form to 
pass messages in.

usually, in my case, if storing data in files, I tend to prefer 
ASCII-based formats.

usually, for passing messages over sockets, I have used "compact" 
specialized binary formats, typically serialized data from some other 
form (such as XML nodes or S-Expressions). although "magic byte value" 
based message formats are initially simpler, they tend to be harder to 
expand later (whereas encoding/decoding some more generic form, though 
initially more effort, can turn out to be easier to maintain and extend 
later).

note: this does not mean SOAP or CORBA or some other "standardized" 
messaging system, rather just that one initially builds and processes 
the messages in some form that is more high-level than spitting out 
bytes, and processing everything via a loop and a big "switch()" or 
similar (although this can be an initially fairly simple option, so has 
some merit due to ease of implementation).


the main reason for picking a binary message-serialization format (for 
something like S-Expressions or XML nodes), would be mostly if there is 
a chance that the serialized data will go over the internet, and a 
textual format can be a bit bulkier (and thus slower to transmit over a 
slower connection), as well as typically being slower to decode (a 
sanely designed message format can be much more quickly unpacked than a 
textual format can be parsed).

sending text over sockets may have merits as well, and is generally 
preferable for "open" protocols.


or such...

[toc] | [prev] | [next] | [standalone]


#11836

FromArved Sandstrom <asandstrom3minus1@eastlink.ca>
Date2012-02-07 20:26 -0400
Message-ID<QijYq.17178$%17.218@newsfe06.iad>
In reply to#11835
On 12-02-07 07:38 PM, BGB wrote:
> On 2/7/2012 11:11 AM, jebblue wrote:
[ SNIP ]
>>
>> I recommend using sockets.
>>
> 
> in general, I agree (sockets generally make the most sense), although
> there are cases where file-based communications can make sense, although
> probably not in the form as described in the OP.
> 
> 
> another issue (besides how to pass messages), is what sort of form to
> pass messages in.
> 
> usually, in my case, if storing data in files, I tend to prefer
> ASCII-based formats.
> 
> usually, for passing messages over sockets, I have used "compact"
> specialized binary formats, typically serialized data from some other
> form (such as XML nodes or S-Expressions). although "magic byte value"
> based message formats are initially simpler, they tend to be harder to
> expand later (whereas encoding/decoding some more generic form, though
> initially more effort, can turn out to be easier to maintain and extend
> later).
> 
> note: this does not mean SOAP or CORBA or some other "standardized"
> messaging system, rather just that one initially builds and processes
> the messages in some form that is more high-level than spitting out
> bytes, and processing everything via a loop and a big "switch()" or
> similar (although this can be an initially fairly simple option, so has
> some merit due to ease of implementation).
> 
> 
> the main reason for picking a binary message-serialization format (for
> something like S-Expressions or XML nodes), would be mostly if there is
> a chance that the serialized data will go over the internet, and a
> textual format can be a bit bulkier (and thus slower to transmit over a
> slower connection), as well as typically being slower to decode (a
> sanely designed message format can be much more quickly unpacked than a
> textual format can be parsed).
> 
> sending text over sockets may have merits as well, and is generally
> preferable for "open" protocols.
> 
> or such...

I've done a fair bit with sockets myself, including recently, in fact
including on a current gig. Some of the message formats have been
designed by others, some by me. A few of them are specialized industry
standards, some are very custom and bespoke.

A few of the formats have been binary: fixed-length blocks of data with
fields at various offsets. Works well enough if it suits the data.

A bunch of others have been text and line-oriented: a fixed number of
lines of data in known order, so that line 10 is always the data for a
particular field.

Other things to consider: JAXB, JSON etc. Minimum coding fuss at the
endpoints if that's what's appropriate for constructing message payloads.

I like text-based protocols, for some simple situations, that behave
like SMTP or POP. But it obviously depends on what you expect your
client and server to do, it's just another approach to be aware of.

One of the big things in designing one's own messaging is error
handling. People generally do just fine with the happy path, but ignore
comprehensive error handling, or get wrapped around the axle trying to
do it.

A lot of situations admit of more than one approach.

AHS
-- 
...wherever the people are well informed they can be trusted with their
own government...
-- Thomas Jefferson, 1789

[toc] | [prev] | [next] | [standalone]


#11846

FromBGB <cr88192@hotmail.com>
Date2012-02-08 01:41 -0700
Message-ID<jgtcid$kfs$1@news.albasani.net>
In reply to#11836
On 2/7/2012 5:26 PM, Arved Sandstrom wrote:
> On 12-02-07 07:38 PM, BGB wrote:
>> On 2/7/2012 11:11 AM, jebblue wrote:
> [ SNIP ]
>>>
>>> I recommend using sockets.
>>>
>>
>> in general, I agree (sockets generally make the most sense), although
>> there are cases where file-based communications can make sense, although
>> probably not in the form as described in the OP.
>>
>>
>> another issue (besides how to pass messages), is what sort of form to
>> pass messages in.
>>
>> usually, in my case, if storing data in files, I tend to prefer
>> ASCII-based formats.
>>
>> usually, for passing messages over sockets, I have used "compact"
>> specialized binary formats, typically serialized data from some other
>> form (such as XML nodes or S-Expressions). although "magic byte value"
>> based message formats are initially simpler, they tend to be harder to
>> expand later (whereas encoding/decoding some more generic form, though
>> initially more effort, can turn out to be easier to maintain and extend
>> later).
>>
>> note: this does not mean SOAP or CORBA or some other "standardized"
>> messaging system, rather just that one initially builds and processes
>> the messages in some form that is more high-level than spitting out
>> bytes, and processing everything via a loop and a big "switch()" or
>> similar (although this can be an initially fairly simple option, so has
>> some merit due to ease of implementation).
>>
>>
>> the main reason for picking a binary message-serialization format (for
>> something like S-Expressions or XML nodes), would be mostly if there is
>> a chance that the serialized data will go over the internet, and a
>> textual format can be a bit bulkier (and thus slower to transmit over a
>> slower connection), as well as typically being slower to decode (a
>> sanely designed message format can be much more quickly unpacked than a
>> textual format can be parsed).
>>
>> sending text over sockets may have merits as well, and is generally
>> preferable for "open" protocols.
>>
>> or such...
>
> I've done a fair bit with sockets myself, including recently, in fact
> including on a current gig. Some of the message formats have been
> designed by others, some by me. A few of them are specialized industry
> standards, some are very custom and bespoke.
>
> A few of the formats have been binary: fixed-length blocks of data with
> fields at various offsets. Works well enough if it suits the data.
>
> A bunch of others have been text and line-oriented: a fixed number of
> lines of data in known order, so that line 10 is always the data for a
> particular field.
>
> Other things to consider: JAXB, JSON etc. Minimum coding fuss at the
> endpoints if that's what's appropriate for constructing message payloads.
>
> I like text-based protocols, for some simple situations, that behave
> like SMTP or POP. But it obviously depends on what you expect your
> client and server to do, it's just another approach to be aware of.
>

well, text need not be all that limiting.
if one has XML or free-form S-Expressions (in their true sense, like in 
Lisp or Scheme, not the mutilated/watered-down Rivest ones), then one 
can do a fair amount with text.

IME, there are many tradeoffs (regarding ease of use, ...) between XML 
and S-Exps, and neither seems "clearly better" (as far as 
representations go, I find S-Exps easier to work with, but namespaces 
and attributes in XML can make it more flexible, as one can more easily 
throw new tags or attributes at the problem with less chance of breaking 
existing code).

an example is this:
<foo> <bar value="3"/> </foo>
and:
(foo (bar 3))

now, consider one wants to add a new field to 'foo' (say 'ln').
<foo ln="15"> <bar value="3"/> </foo>
and:
(foo 15 (bar 3))

a difference here is that existing code will probably not even notice 
the new XML attribute, whereas the positional nature of most 
S-Expressions makes the latter far more likely to break something (and 
there is no good way to "annotate" an S-Exp, whereas with XML it is 
fairly solidly defined that one can simply add new attributes).


note: my main way of working with XML is typically via DOM-style 
interfaces (if I am using it, it is typically because I am directly 
working with the data structure, and not as the result of some dumb-ass 
"data binding" crud...).


typically, the "internal representation" and "concrete serialization" 
are different:
I may use a textual XML serialization, or just as easily, I could use a 
binary format;
likewise for S-Exps (actually, I probably far more often represent 
S-Exps as a binary format of one form or another than I use them in a 
form externally serialized as text).

all hail the mighty DOM-node or CONS-cell...


> One of the big things in designing one's own messaging is error
> handling. People generally do just fine with the happy path, but ignore
> comprehensive error handling, or get wrapped around the axle trying to
> do it.
>

yeah, but this applies to programming in general, so message-passing is 
likely nothing special here. one issue maybe special to sockets though 
is the matter of whether or not the whole message has been received, 
often resulting in some annoying code to basically read messages from 
the socket and not decode them until the entire message has been received.


> A lot of situations admit of more than one approach.
>

agreed.

it is like me and file-formats.
often I just use ASCII text (simple, easy, editable in Notepad or 
similar, ...).

I make plenty of use of simple line-oriented text formats as well.

other times, I might use more advanced binary formats, or maybe even 
employ the use of "data compression" techniques (such as Huffman 
coding), so a lot depends.

[toc] | [prev] | [next] | [standalone]


#11851

FromArved Sandstrom <asandstrom3minus1@eastlink.ca>
Date2012-02-08 07:19 -0400
Message-ID<JSsYq.10041$W87.8642@newsfe02.iad>
In reply to#11846
On 12-02-08 04:41 AM, BGB wrote:
> On 2/7/2012 5:26 PM, Arved Sandstrom wrote:
>> On 12-02-07 07:38 PM, BGB wrote:
>>> On 2/7/2012 11:11 AM, jebblue wrote:
>> [ SNIP ]
>>>>
>>>> I recommend using sockets.
>>>>
>>>
>>> in general, I agree (sockets generally make the most sense), although
>>> there are cases where file-based communications can make sense, although
>>> probably not in the form as described in the OP.
>>>
>>>
>>> another issue (besides how to pass messages), is what sort of form to
>>> pass messages in.
>>>
>>> usually, in my case, if storing data in files, I tend to prefer
>>> ASCII-based formats.
>>>
>>> usually, for passing messages over sockets, I have used "compact"
>>> specialized binary formats, typically serialized data from some other
>>> form (such as XML nodes or S-Expressions). although "magic byte value"
>>> based message formats are initially simpler, they tend to be harder to
>>> expand later (whereas encoding/decoding some more generic form, though
>>> initially more effort, can turn out to be easier to maintain and extend
>>> later).
>>>
>>> note: this does not mean SOAP or CORBA or some other "standardized"
>>> messaging system, rather just that one initially builds and processes
>>> the messages in some form that is more high-level than spitting out
>>> bytes, and processing everything via a loop and a big "switch()" or
>>> similar (although this can be an initially fairly simple option, so has
>>> some merit due to ease of implementation).
>>>
>>>
>>> the main reason for picking a binary message-serialization format (for
>>> something like S-Expressions or XML nodes), would be mostly if there is
>>> a chance that the serialized data will go over the internet, and a
>>> textual format can be a bit bulkier (and thus slower to transmit over a
>>> slower connection), as well as typically being slower to decode (a
>>> sanely designed message format can be much more quickly unpacked than a
>>> textual format can be parsed).
>>>
>>> sending text over sockets may have merits as well, and is generally
>>> preferable for "open" protocols.
>>>
>>> or such...
>>
>> I've done a fair bit with sockets myself, including recently, in fact
>> including on a current gig. Some of the message formats have been
>> designed by others, some by me. A few of them are specialized industry
>> standards, some are very custom and bespoke.
>>
>> A few of the formats have been binary: fixed-length blocks of data with
>> fields at various offsets. Works well enough if it suits the data.
>>
>> A bunch of others have been text and line-oriented: a fixed number of
>> lines of data in known order, so that line 10 is always the data for a
>> particular field.
>>
>> Other things to consider: JAXB, JSON etc. Minimum coding fuss at the
>> endpoints if that's what's appropriate for constructing message payloads.
>>
>> I like text-based protocols, for some simple situations, that behave
>> like SMTP or POP. But it obviously depends on what you expect your
>> client and server to do, it's just another approach to be aware of.
>>
> 
> well, text need not be all that limiting.

You may have misunderstood something I said if you got that impression
from me, that text is all that limiting. :-)

[ SNIP ]

> note: my main way of working with XML is typically via DOM-style
> interfaces (if I am using it, it is typically because I am directly
> working with the data structure, and not as the result of some dumb-ass
> "data binding" crud...).

I haven't been able to completely avoid using the DOM, but I loathe the
API. If I'm using XML at all, and JAXB suits, I'll use JAXB. More
generally I'll use SAX or StAX.

I almost never encounter a situation where DOM is called for, simply
because no random access to the document is called for. When I send XML
back and forth as a payload, the entire thing is meant to be used, and
it makes sense to do the immediate and complete conversion into real
information rather than storing it into an opaque and kludgy DOM
representation.

For a lot of situations, not just message passing between endpoints, I
have backed away from XML anyway. For configuration files I have gotten
newly enthused by .properties files, because so often they fit the bill
much better than XML configuration files. And I mentioned JSON
previously, I prefer that to XML in many situations now.

[ SNIP ]

>> One of the big things in designing one's own messaging is error
>> handling. People generally do just fine with the happy path, but ignore
>> comprehensive error handling, or get wrapped around the axle trying to
>> do it.
> 
> yeah, but this applies to programming in general, so message-passing is
> likely nothing special here.

That's true, but it's maybe a bit more of an art form with messages.
Your message producer may be Java and produce beautiful exceptions in
your carefully designed exception hierarchy, but your clients may very
well not be Java at all, in which case you may end up with an error
message sub-protocol that borrows ideas from from HTTP status codes.

A lot of Java programmers these days maybe have never really dealt with
return codes, because we sort of tell them not to use them in Java, but
in the case of implementation-neutral status codes (including ones for
errors) that's really the design mindset that you need to be in: status
codes.

one issue maybe special to sockets though
> is the matter of whether or not the whole message has been received,
> often resulting in some annoying code to basically read messages from
> the socket and not decode them until the entire message has been received.

There is that. Although I find that once you've worked through one or
two socket implementations that you tend to devise some pretty re-usable
code for handling the incomplete message situations.
[ SNIP ]

AHS
-- 
...wherever the people are well informed they can be trusted with their
own government...
-- Thomas Jefferson, 1789

[toc] | [prev] | [next] | [standalone]


#11856

FromBGB <cr88192@hotmail.com>
Date2012-02-08 12:07 -0700
Message-ID<jguh6l$a3o$1@news.albasani.net>
In reply to#11851
On 2/8/2012 4:19 AM, Arved Sandstrom wrote:
> On 12-02-08 04:41 AM, BGB wrote:
>> On 2/7/2012 5:26 PM, Arved Sandstrom wrote:
>>> On 12-02-07 07:38 PM, BGB wrote:
>>>> On 2/7/2012 11:11 AM, jebblue wrote:
>>> [ SNIP ]
>>>>>
>>>>> I recommend using sockets.
>>>>>
>>>>
>>>> in general, I agree (sockets generally make the most sense), although
>>>> there are cases where file-based communications can make sense, although
>>>> probably not in the form as described in the OP.
>>>>
>>>>
>>>> another issue (besides how to pass messages), is what sort of form to
>>>> pass messages in.
>>>>
>>>> usually, in my case, if storing data in files, I tend to prefer
>>>> ASCII-based formats.
>>>>
>>>> usually, for passing messages over sockets, I have used "compact"
>>>> specialized binary formats, typically serialized data from some other
>>>> form (such as XML nodes or S-Expressions). although "magic byte value"
>>>> based message formats are initially simpler, they tend to be harder to
>>>> expand later (whereas encoding/decoding some more generic form, though
>>>> initially more effort, can turn out to be easier to maintain and extend
>>>> later).
>>>>
>>>> note: this does not mean SOAP or CORBA or some other "standardized"
>>>> messaging system, rather just that one initially builds and processes
>>>> the messages in some form that is more high-level than spitting out
>>>> bytes, and processing everything via a loop and a big "switch()" or
>>>> similar (although this can be an initially fairly simple option, so has
>>>> some merit due to ease of implementation).
>>>>
>>>>
>>>> the main reason for picking a binary message-serialization format (for
>>>> something like S-Expressions or XML nodes), would be mostly if there is
>>>> a chance that the serialized data will go over the internet, and a
>>>> textual format can be a bit bulkier (and thus slower to transmit over a
>>>> slower connection), as well as typically being slower to decode (a
>>>> sanely designed message format can be much more quickly unpacked than a
>>>> textual format can be parsed).
>>>>
>>>> sending text over sockets may have merits as well, and is generally
>>>> preferable for "open" protocols.
>>>>
>>>> or such...
>>>
>>> I've done a fair bit with sockets myself, including recently, in fact
>>> including on a current gig. Some of the message formats have been
>>> designed by others, some by me. A few of them are specialized industry
>>> standards, some are very custom and bespoke.
>>>
>>> A few of the formats have been binary: fixed-length blocks of data with
>>> fields at various offsets. Works well enough if it suits the data.
>>>
>>> A bunch of others have been text and line-oriented: a fixed number of
>>> lines of data in known order, so that line 10 is always the data for a
>>> particular field.
>>>
>>> Other things to consider: JAXB, JSON etc. Minimum coding fuss at the
>>> endpoints if that's what's appropriate for constructing message payloads.
>>>
>>> I like text-based protocols, for some simple situations, that behave
>>> like SMTP or POP. But it obviously depends on what you expect your
>>> client and server to do, it's just another approach to be aware of.
>>>
>>
>> well, text need not be all that limiting.
>
> You may have misunderstood something I said if you got that impression
> from me, that text is all that limiting. :-)
>
> [ SNIP ]
>

ok.

it came off that you were implying that text only really worked well for 
simple protocols, like SMTP, POP, HTTP, ...


>> note: my main way of working with XML is typically via DOM-style
>> interfaces (if I am using it, it is typically because I am directly
>> working with the data structure, and not as the result of some dumb-ass
>> "data binding" crud...).
>
> I haven't been able to completely avoid using the DOM, but I loathe the
> API. If I'm using XML at all, and JAXB suits, I'll use JAXB. More
> generally I'll use SAX or StAX.
>

I have rarely done things for which SAX has made sense...
usually in cases where SAX would make sense, I end up using 
line-oriented text formats instead (because there is often little 
obvious reason for why XML syntax would make much sense).


> I almost never encounter a situation where DOM is called for, simply
> because no random access to the document is called for. When I send XML
> back and forth as a payload, the entire thing is meant to be used, and
> it makes sense to do the immediate and complete conversion into real
> information rather than storing it into an opaque and kludgy DOM
> representation.
>

often, I use it for things like compiler ASTs, where it competes some 
against S-Expressions (they are produced by the main parser, worked on, 
and then later converted into bytecode or similar).

typically, one works by walking the tree, and potentially 
rebuilding/rewriting a new tree in the process, or maybe adding 
annotations to the existing tree.


a recent case where I did consider using XML as a message-passing 
protocol, I ended up opting for S-Expressions (or, more properly, 
Lisp-style lists) instead, mostly because they are a lot easier to build 
and process, and much less painful than working with a DOM-style API 
(and also because S-Expressions tend to perform better and use less 
memory in my case as well...).

typically, the messages are tree-structured data of some sort (in the 
recent example, it was being used for scene-graph delta messages, which 
basically update the status of various objects in the scene, as well as 
passing other events for things "going on", like sound-effects being 
heard, updates to camera location and status, ...).


it is also desirable to keep the serialized representation small, since 
a lot may be going on (in real time), and it would be annoying (say, to 
players) if the connection got needlessly bogged down sending lots of 
overly verbose update messages (more so if one has stuff like 
network-synchronized rag-dolls or similar, where a ragdoll may send 
position updates for nearly every bone for every frame).

say:
(bonedelta 499 (bone 0 (org ...) (rot ...)) (bone 1 (org ...) (rot ...)) 
...)
(bonedelta 515 ...)
...


hence, it may make a little sense to employ a compressed binary format.
I also personally dislike schemas or similar concepts, as they tend to 
make things brittle (both the transmitter and receiver need a correct 
and up-to-date schema, creating a higher risk of version issues), and 
typically don't really compress all that much better (and are 
potentially worse) than what a decent adaptive coding can do.

("on the wire", S-Exps and XML are not all that drastically different, 
the main practical differences are more in terms of how one may work 
with them in-program).


granted, yes, text+deflate also works OK if one is feeling lazy (since 
IME Deflate will typically reduce textual XML or S-Exps to around 
10%-25% their original size, vs say a 5%-10% one might get with a 
specialized binary format).

there is also the tradeoff of designing a binary format to be standalone 
(say, including its own Huffman compressor), or to be used in 
combination with deflate (at which point one tries to design the format 
to instead produce data which deflate can utilize efficiently).

in the latter option, there is the secondary concern of external deflate 
(assuming that the data will probably be sent in a compressed channel or 
stored in a ZIP file or similar), or using deflate internally (like in 
PNG or similar).

there are many tradeoffs...


> For a lot of situations, not just message passing between endpoints, I
> have backed away from XML anyway. For configuration files I have gotten
> newly enthused by .properties files, because so often they fit the bill
> much better than XML configuration files. And I mentioned JSON
> previously, I prefer that to XML in many situations now.
>

I typically use line-oriented text formats for most of these purposes...

never really did understand why someone would use XML for things like 
configuration files (it neither makes them easier to process, nor does 
it help anything with users trying to edit them).


as-is, my configuration format consists of "console commands", which may 
in turn set "cvars" or issue key-binding commands, ...

for another (more serious) system, I am using a format which is 
partially a hybrid of INI and REG files (it is for a registry-like 
hierarchical database). I have on/off considered switching to a binary 
database format, but never got around to it.

some amount of other data is stored in formats similar to the Quake map 
format, or other special-purpose text formats.


> [ SNIP ]
>
>>> One of the big things in designing one's own messaging is error
>>> handling. People generally do just fine with the happy path, but ignore
>>> comprehensive error handling, or get wrapped around the axle trying to
>>> do it.
>>
>> yeah, but this applies to programming in general, so message-passing is
>> likely nothing special here.
>
> That's true, but it's maybe a bit more of an art form with messages.
> Your message producer may be Java and produce beautiful exceptions in
> your carefully designed exception hierarchy, but your clients may very
> well not be Java at all, in which case you may end up with an error
> message sub-protocol that borrows ideas from from HTTP status codes.
>
> A lot of Java programmers these days maybe have never really dealt with
> return codes, because we sort of tell them not to use them in Java, but
> in the case of implementation-neutral status codes (including ones for
> errors) that's really the design mindset that you need to be in: status
> codes.
>

granted, I am actually primarily a C and C++ programmer, but 
message-passing isn't particularly language-specific. granted, yes, the 
lack of "standard" exceptions is an annoyance in C, where typically one 
either needs to not use exceptions, or end up using non-portable 
exception mechanisms, and there is no particularly good way to "build 
ones' own", although some people have before done some fairly "creative" 
things with macros...


> one issue maybe special to sockets though
>> is the matter of whether or not the whole message has been received,
>> often resulting in some annoying code to basically read messages from
>> the socket and not decode them until the entire message has been received.
>
> There is that. Although I find that once you've worked through one or
> two socket implementations that you tend to devise some pretty re-usable
> code for handling the incomplete message situations.
> [ SNIP ]
>

yep.

one can always tag messages and then give them with a length.

{ tag, length, data[length] }
message is then not processed until entire data region is received.
typically, this is plenty sufficient.


likewise, a PPP/HDLC style system (message start/end codes) could also 
be used.


depending on other factors, one can also do things like in JPEG or MPEG, 
and use a special escape-code for messages and control-codes.

this can allow a top-level message format like:
{ escape-code, tag [ length, data[length] ... ] }

typically, in such cases (I have seen) there have been ways to escape 
the escape-code, usually for cases where the escape code appeared 
by-chance in the data. this in-turn adds the annoyance of typically 
having to escape any escape-codes in the payload data.

some others have partly worked around the above by making the escape 
code fairly long (32 or 48 bits or more) and very unlikely to appear by 
chance, and likely involving "sanity checks" to try to rule out false 
positives.

say: { escape-magic, tag, length, data[length], checksum }
with the assumption that chance is very unlikely to lead to all of:
an escape magic, a valid tag value, a sane length, and a valid checksum.

depending, the escape-magic and tag can be the same value.

for example:
the byte 0x7E is magic;
7E,00 escapes 7E (or maybe 7E,7E)
7E,01 Start Of Message (followed by message data)
7E,02 End Of Message (maybe, followed by checksum)
others: reserved for link-control messages.


then one can pass encoded messages over the link.

typically, I have not tried parsing incomplete messages, as trying to 
make a message decoder deal gracefully with truncated data is a bit more 
of a hassle.

depending on other factors (say, if one is using Huffman), then one can 
also use special markers to transmit the Huffman tables and other things.

say:
7E,03: Stream Reset (possibly followed by a stream/protocol ID magic)
7E,04-07: Huffman Tables 0-3
7E,08: End Of Huffman Table
...

[toc] | [prev] | [next] | [standalone]


#11864

FromArne Vajhøj <arne@vajhoej.dk>
Date2012-02-08 21:16 -0500
Message-ID<4f332c6d$0$288$14726298@news.sunsite.dk>
In reply to#11856
On 2/8/2012 2:07 PM, BGB wrote:
> On 2/8/2012 4:19 AM, Arved Sandstrom wrote:
>> On 12-02-08 04:41 AM, BGB wrote:
>>> note: my main way of working with XML is typically via DOM-style
>>> interfaces (if I am using it, it is typically because I am directly
>>> working with the data structure, and not as the result of some dumb-ass
>>> "data binding" crud...).
>>
>> I haven't been able to completely avoid using the DOM, but I loathe the
>> API. If I'm using XML at all, and JAXB suits, I'll use JAXB. More
>> generally I'll use SAX or StAX.
>>
>
> I have rarely done things for which SAX has made sense...
> usually in cases where SAX would make sense, I end up using
> line-oriented text formats instead (because there is often little
> obvious reason for why XML syntax would make much sense).

Non flat structure and validation comes to mind.

Arne

[toc] | [prev] | [next] | [standalone]


#11865

FromBGB <cr88192@hotmail.com>
Date2012-02-08 19:50 -0700
Message-ID<jgvcb9$dvi$1@news.albasani.net>
In reply to#11864
On 2/8/2012 7:16 PM, Arne Vajhøj wrote:
> On 2/8/2012 2:07 PM, BGB wrote:
>> On 2/8/2012 4:19 AM, Arved Sandstrom wrote:
>>> On 12-02-08 04:41 AM, BGB wrote:
>>>> note: my main way of working with XML is typically via DOM-style
>>>> interfaces (if I am using it, it is typically because I am directly
>>>> working with the data structure, and not as the result of some dumb-ass
>>>> "data binding" crud...).
>>>
>>> I haven't been able to completely avoid using the DOM, but I loathe the
>>> API. If I'm using XML at all, and JAXB suits, I'll use JAXB. More
>>> generally I'll use SAX or StAX.
>>>
>>
>> I have rarely done things for which SAX has made sense...
>> usually in cases where SAX would make sense, I end up using
>> line-oriented text formats instead (because there is often little
>> obvious reason for why XML syntax would make much sense).
>
> Non flat structure and validation comes to mind.
>

fair enough.

often, one can implement non-flat structures with line-oriented formats, 
for example:
...
groupDef {
...
groupDef {
itemDef {
...
}
...
}
...
}

a lot of time this may be combined with cosmetic indentation, but this 
does not change if it is a line-oriented format, for example, writing:
groupDef
{
...
}

could very-well break the parser.


typically, I have not used validation:
if there is anything to validate, typically this logic will be placed in 
the logic to parse the text.

a lot of times, code operates under the assumption that nearly anything 
which can be reasonably done is valid de-facto (the code is written, 
however, to ideally not do anything compromising).


granted, typically I don't deal a whole lot with anything "security 
critical" or where there is much need to worry about "trust" or 
"authorization" or similar (or if privacy or money or similar was 
involved...). maybe if security were more of a concern, then added 
layers of pedantics and validation would make a lot more sense.

in my typical use-cases, the theoretical worst case would probably be if 
a 3rd party could somehow break the app and get control of the users' OS 
or similar and cause damage, but again, modern Windows is itself partly 
designed to try to defend against this (running applications by default 
with constrained privileges, ...).

[toc] | [prev] | [next] | [standalone]


#11875

FromArved Sandstrom <asandstrom3minus1@eastlink.ca>
Date2012-02-09 06:24 -0400
Message-ID<l9NYq.10840$EF2.8233@newsfe18.iad>
In reply to#11865
On 12-02-08 10:50 PM, BGB wrote:
> On 2/8/2012 7:16 PM, Arne Vajhøj wrote:
>> On 2/8/2012 2:07 PM, BGB wrote:
>>> On 2/8/2012 4:19 AM, Arved Sandstrom wrote:
>>>> On 12-02-08 04:41 AM, BGB wrote:
>>>>> note: my main way of working with XML is typically via DOM-style
>>>>> interfaces (if I am using it, it is typically because I am directly
>>>>> working with the data structure, and not as the result of some
>>>>> dumb-ass
>>>>> "data binding" crud...).
>>>>
>>>> I haven't been able to completely avoid using the DOM, but I loathe the
>>>> API. If I'm using XML at all, and JAXB suits, I'll use JAXB. More
>>>> generally I'll use SAX or StAX.
>>>>
>>>
>>> I have rarely done things for which SAX has made sense...
>>> usually in cases where SAX would make sense, I end up using
>>> line-oriented text formats instead (because there is often little
>>> obvious reason for why XML syntax would make much sense).
>>
>> Non flat structure and validation comes to mind.
>>
> 
> fair enough.
> 
> often, one can implement non-flat structures with line-oriented formats,
> for example:
> ...
> groupDef {
> ...
> groupDef {
> itemDef {
> ...
> }
> ...
> }
> ...
> }
[ SNIP ]

No need for the braces, if you're going to use those all you gain over
the XML is terseness.

Consider line-oriented files/messages like .properties files: these can
describe hierarchical structures perfectly well if you've got an
understood key=value syntax, specifically with a hierarchy-supporting
syntax for the keys. Easy to read and edit, easy to parse.

As an example take a look at log4j .properties and XML configuration
files. All you gain with the XML is the ability to validate against a
log4j DTD.

> a lot of times, code operates under the assumption that nearly anything
> which can be reasonably done is valid de-facto (the code is written,
> however, to ideally not do anything compromising).
> 
> granted, typically I don't deal a whole lot with anything "security
> critical" or where there is much need to worry about "trust" or
> "authorization" or similar (or if privacy or money or similar was
> involved...). maybe if security were more of a concern, then added
> layers of pedantics and validation would make a lot more sense.
> 
> in my typical use-cases, the theoretical worst case would probably be if
> a 3rd party could somehow break the app and get control of the users' OS
> or similar and cause damage, but again, modern Windows is itself partly
> designed to try to defend against this (running applications by default
> with constrained privileges, ...).
> 
This is a narrow view of application security. Unless you're writing toy
apps, one would expect that your apps are doing *something*, and that
something includes access to databases or files or other resources.
Furthermore, if your app is used by anyone other than yourself, another
asset is in play, and that's your personal, team's or business's
reputation.

Privacy-sensitive data, or financial data, doesn't have to be involved,
and you don't need the actions of a malicious third party, in order to
have an application security problem. If your code is such that it
corrupts any persistent data, say, or is seriously under-performant
under load, or intermittently breaks and the app has to be re-started,
you've managed to trample all over the Integrity [1] and Availability
security attributes of CAI (Confidentiality, Availability,
Integrity)...all without the help of any malicious external threats.

Do you think your users care who or what mangled part of the
organizational data, or who or what is responsible for 20 percent
downtime? Some of your stakeholders will, sure, when culprits are being
sought, but most of your users will just care about proper function.

All application security starts with good coding. That's why so much of
standards like the Java Secure Coding Guidelines, or OWASP
Development/Code Review/Testing guides, have to do with good coding. And
I don't believe you can really relax your standards with some apps and
have high standards in another.

AHS

1. Strictly speaking not an integrity violation if you can detect the
unintended data corruption, ideally know what caused it, and even better
repair it, but in practice once the damage is done you often
*effectively* can't easily recover; the effort of detecting and fixing
is itself punitive.
-- 
...wherever the people are well informed they can be trusted with their
own government...
-- Thomas Jefferson, 1789

[toc] | [prev] | [next] | [standalone]


#11878

FromBGB <cr88192@hotmail.com>
Date2012-02-09 09:15 -0700
Message-ID<jh0rgm$hn9$1@news.albasani.net>
In reply to#11875
On 2/9/2012 3:24 AM, Arved Sandstrom wrote:
> On 12-02-08 10:50 PM, BGB wrote:
>> On 2/8/2012 7:16 PM, Arne Vajhøj wrote:
>>> On 2/8/2012 2:07 PM, BGB wrote:
>>>> On 2/8/2012 4:19 AM, Arved Sandstrom wrote:
>>>>> On 12-02-08 04:41 AM, BGB wrote:
>>>>>> note: my main way of working with XML is typically via DOM-style
>>>>>> interfaces (if I am using it, it is typically because I am directly
>>>>>> working with the data structure, and not as the result of some
>>>>>> dumb-ass
>>>>>> "data binding" crud...).
>>>>>
>>>>> I haven't been able to completely avoid using the DOM, but I loathe the
>>>>> API. If I'm using XML at all, and JAXB suits, I'll use JAXB. More
>>>>> generally I'll use SAX or StAX.
>>>>>
>>>>
>>>> I have rarely done things for which SAX has made sense...
>>>> usually in cases where SAX would make sense, I end up using
>>>> line-oriented text formats instead (because there is often little
>>>> obvious reason for why XML syntax would make much sense).
>>>
>>> Non flat structure and validation comes to mind.
>>>
>>
>> fair enough.
>>
>> often, one can implement non-flat structures with line-oriented formats,
>> for example:
>> ...
>> groupDef {
>> ...
>> groupDef {
>> itemDef {
>> ...
>> }
>> ...
>> }
>> ...
>> }
> [ SNIP ]
>
> No need for the braces, if you're going to use those all you gain over
> the XML is terseness.
>

well, if the format is still line-oriented, one can still parse the 
files using a loop, getting and splitting strings, and checking the 
first token of each line.

parsing XML is a little more invovlved, since:
items may be split across lines, or multiple items may exist on the same 
line;
one can no longer use whitespace or commas as the primary deliminator;
...

granted, yes, one can use SAX or similar, but alas...

one can wonder though, what really would be the gain of using XML syntax 
in many such cases, vs the typical "relative niceness" of a line 
oriented format.

like, say I have a format which looks like:
{
"classname" "func_door"
"angle" "-1"
...
{
[  1  0  0 16 ] brick/mybrick [ 0 1 0 0 ] [ 0 0 1 0 ]
[ -1  0  0 16 ] brick/mybrick [ 0 1 0 0 ] [ 0 0 1 0 ]
[  0  1  0 16 ] brick/mybrick [ 1 0 0 0 ] [ 0 0 1 0 ]
[  0 -1  0 16 ] brick/mybrick [ 1 0 0 0 ] [ 0 0 1 0 ]
[  0  0  1 16 ] brick/mybrick [ 1 0 0 0 ] [ 0 1 0 0 ]
[  0  0 -1 16 ] brick/mybrick [ 1 0 0 0 ] [ 0 1 0 0 ]
}
}

would it really look much better as:
<entity>
<field var="classname" value="func_door"/>
<field var="angle" value="-1"/>
...
<brush>
<face plane="1 0 0 16" texture="brick/mybrick" sdir="0 1 0 0" tdir="0 0 
1 0"/>
...
</brush>
</entity>

even despite the parser being more generic, and it being better labeled 
what everything is, is it really an improvement WRT, say, readability?...


> Consider line-oriented files/messages like .properties files: these can
> describe hierarchical structures perfectly well if you've got an
> understood key=value syntax, specifically with a hierarchy-supporting
> syntax for the keys. Easy to read and edit, easy to parse.
>

yes, but this defeats your own prior point, namely indirectly asserting 
that line-oriented == flat-structure.

point is, one can have hierarchical line-oriented files.


> As an example take a look at log4j .properties and XML configuration
> files. All you gain with the XML is the ability to validate against a
> log4j DTD.
>
>> a lot of times, code operates under the assumption that nearly anything
>> which can be reasonably done is valid de-facto (the code is written,
>> however, to ideally not do anything compromising).
>>
>> granted, typically I don't deal a whole lot with anything "security
>> critical" or where there is much need to worry about "trust" or
>> "authorization" or similar (or if privacy or money or similar was
>> involved...). maybe if security were more of a concern, then added
>> layers of pedantics and validation would make a lot more sense.
>>
>> in my typical use-cases, the theoretical worst case would probably be if
>> a 3rd party could somehow break the app and get control of the users' OS
>> or similar and cause damage, but again, modern Windows is itself partly
>> designed to try to defend against this (running applications by default
>> with constrained privileges, ...).
>>
> This is a narrow view of application security. Unless you're writing toy
> apps, one would expect that your apps are doing *something*, and that
> something includes access to databases or files or other resources.
> Furthermore, if your app is used by anyone other than yourself, another
> asset is in play, and that's your personal, team's or business's
> reputation.
>

"someone steals' the user's save-games!", that would be scary, or not 
really...

most of the files in a game are generic resource data, but stealing them 
is of little concern, and damaging them is more likely to be an 
annoyance than an actual threat "oh crap, I might have to reinstall...".


> Privacy-sensitive data, or financial data, doesn't have to be involved,
> and you don't need the actions of a malicious third party, in order to
> have an application security problem. If your code is such that it
> corrupts any persistent data, say, or is seriously under-performant
> under load, or intermittently breaks and the app has to be re-started,
> you've managed to trample all over the Integrity [1] and Availability
> security attributes of CAI (Confidentiality, Availability,
> Integrity)...all without the help of any malicious external threats.
>

typically, crashes are more an annoyance than a major threat.

consider Skyrim: the damn thing can't usually keep going for more than 1 
or 2 hours before crashing-to-desktop or similar.

of course, not everyone aspires towards Bethesda levels of stability.


> Do you think your users care who or what mangled part of the
> organizational data, or who or what is responsible for 20 percent
> downtime? Some of your stakeholders will, sure, when culprits are being
> sought, but most of your users will just care about proper function.
>

only likely matters if it is some sort of server-based or business type app.

ok, a game-server crashing could be a bit annoying if one were making 
something like an MMORPG or something (like WoW...).


in my case, I am not:
the online play would likely be more for things like user-run deathmatch 
servers and similar.


> All application security starts with good coding. That's why so much of
> standards like the Java Secure Coding Guidelines, or OWASP
> Development/Code Review/Testing guides, have to do with good coding. And
> I don't believe you can really relax your standards with some apps and
> have high standards in another.
>

it is more a matter of productivity:
focus on security, code-quality, ... in places where it is important;
otherwise, whatever one can mash together which basically works is 
arguably good enough.

granted, it is not like there aren't some things I care about, like I 
prefer clean and nice code over a tangled mess, but ultimately this may 
be secondary to the greater concern, "get it done" (as, what good is 
good code if the product can never get out the door and on the market?).


it is like with art:
some people can be perfectionist, and worry about tiny details which 
hardly anyone would ever notice;
other people can try to make something "good enough" and hope users 
don't notice or care about any little graphical imperfections.


> AHS
>
> 1. Strictly speaking not an integrity violation if you can detect the
> unintended data corruption, ideally know what caused it, and even better
> repair it, but in practice once the damage is done you often
> *effectively* can't easily recover; the effort of detecting and fixing
> is itself punitive.

potentially, but it depends on the relative costs.

if the worst case is forcing a reinstall, this is much less of an issue 
than, say, if it breaks their savegames, which is much less of an issue 
than if any "actually important" data is involved (compromises users' 
privacy or security, causes damage to their computer, ...).

say, one doesn't want to have their app be a vector for virus delivery, 
as this can give a bad reputation.


but, alas...

[toc] | [prev] | [next] | [standalone]


#11886

FromArved Sandstrom <asandstrom3minus1@eastlink.ca>
Date2012-02-09 18:58 -0400
Message-ID<lcYYq.12118$EF2.10458@newsfe18.iad>
In reply to#11878
On 12-02-09 12:15 PM, BGB wrote:
> On 2/9/2012 3:24 AM, Arved Sandstrom wrote:
[ SNIP ]
> 
>> Consider line-oriented files/messages like .properties files: these can
>> describe hierarchical structures perfectly well if you've got an
>> understood key=value syntax, specifically with a hierarchy-supporting
>> syntax for the keys. Easy to read and edit, easy to parse.
> 
> yes, but this defeats your own prior point, namely indirectly asserting
> that line-oriented == flat-structure.

Minor quibble, I didn't make such a point, not even indirectly. You may
be confusing me with Arne.

> point is, one can have hierarchical line-oriented files.
[ SNIP ]

Yes.

AHS
-- 
...wherever the people are well informed they can be trusted with their
own government...
-- Thomas Jefferson, 1789

[toc] | [prev] | [next] | [standalone]


#11887

FromBGB <cr88192@hotmail.com>
Date2012-02-09 16:15 -0700
Message-ID<jh1k4f$anj$1@news.albasani.net>
In reply to#11886
On 2/9/2012 3:58 PM, Arved Sandstrom wrote:
> On 12-02-09 12:15 PM, BGB wrote:
>> On 2/9/2012 3:24 AM, Arved Sandstrom wrote:
> [ SNIP ]
>>
>>> Consider line-oriented files/messages like .properties files: these can
>>> describe hierarchical structures perfectly well if you've got an
>>> understood key=value syntax, specifically with a hierarchy-supporting
>>> syntax for the keys. Easy to read and edit, easy to parse.
>>
>> yes, but this defeats your own prior point, namely indirectly asserting
>> that line-oriented == flat-structure.
>
> Minor quibble, I didn't make such a point, not even indirectly. You may
> be confusing me with Arne.
>

ok. both names started with 'Ar', so I guess I didn't notice the change...

>> point is, one can have hierarchical line-oriented files.
> [ SNIP ]
>
> Yes.
>
> AHS

[toc] | [prev] | [next] | [standalone]


#11891

FromArne Vajhøj <arne@vajhoej.dk>
Date2012-02-09 18:50 -0500
Message-ID<4f345bcb$0$291$14726298@news.sunsite.dk>
In reply to#11886
On 2/9/2012 5:58 PM, Arved Sandstrom wrote:
> On 12-02-09 12:15 PM, BGB wrote:
>> On 2/9/2012 3:24 AM, Arved Sandstrom wrote:
> [ SNIP ]
>>
>>> Consider line-oriented files/messages like .properties files: these can
>>> describe hierarchical structures perfectly well if you've got an
>>> understood key=value syntax, specifically with a hierarchy-supporting
>>> syntax for the keys. Easy to read and edit, easy to parse.
>>
>> yes, but this defeats your own prior point, namely indirectly asserting
>> that line-oriented == flat-structure.
>
> Minor quibble, I didn't make such a point, not even indirectly. You may
> be confusing me with Arne.
>
>> point is, one can have hierarchical line-oriented files.
> [ SNIP ]
>
> Yes.

You can have non flat structures other than XML, but parsing
quickly becomes very complex.

Arne

[toc] | [prev] | [next] | [standalone]


#11900

FromBGB <cr88192@hotmail.com>
Date2012-02-09 21:40 -0700
Message-ID<jh275i$kf6$1@news.albasani.net>
In reply to#11891
On 2/9/2012 4:50 PM, Arne Vajhøj wrote:
> On 2/9/2012 5:58 PM, Arved Sandstrom wrote:
>> On 12-02-09 12:15 PM, BGB wrote:
>>> On 2/9/2012 3:24 AM, Arved Sandstrom wrote:
>> [ SNIP ]
>>>
>>>> Consider line-oriented files/messages like .properties files: these can
>>>> describe hierarchical structures perfectly well if you've got an
>>>> understood key=value syntax, specifically with a hierarchy-supporting
>>>> syntax for the keys. Easy to read and edit, easy to parse.
>>>
>>> yes, but this defeats your own prior point, namely indirectly asserting
>>> that line-oriented == flat-structure.
>>
>> Minor quibble, I didn't make such a point, not even indirectly. You may
>> be confusing me with Arne.
>>
>>> point is, one can have hierarchical line-oriented files.
>> [ SNIP ]
>>
>> Yes.
>
> You can have non flat structures other than XML, but parsing
> quickly becomes very complex.
>

IMO: not particularly...


tree-structured line-oriented text formats are not that much more 
complicated to parse than flat-list line-oriented text formats (the only 
obvious difference is that rather than a single loop, one typically has 
multiple loops).

it all still basically amounts to:
read a line;
split the string;
do something with the split strings.


the main hassle IMO is when one needs to go from line-oriented to 
token-oriented formats (say, writing a parser for something like 
S-Expressions or XML). then one will typically need to buffer in the 
whole file and read out a token at a time.

the next big complexity jump is when one starts dealing with things like 
operator precedence, say, when parsing typical programming-language 
style syntax (say, Java or JavaScript like).

after that, is probably context-sensitive declarations (namely, like in 
C or C++), at which point parsing starts becoming a good deal more of a 
PITA (writing a C++ parser is probably "non-trivial", since many cases 
are ambiguous and one can often only know the correct way to parse 
something by matching it against prior declarations).


personally, I like recursive hand-written descent parsing, as it is 
fairly straightforwards and doesn't depend on external tools.

[toc] | [prev] | [next] | [standalone]


#11937

FromArne Vajhøj <arne@vajhoej.dk>
Date2012-02-11 14:47 -0500
Message-ID<4f36c5ec$0$294$14726298@news.sunsite.dk>
In reply to#11900
On 2/9/2012 11:40 PM, BGB wrote:
> On 2/9/2012 4:50 PM, Arne Vajhøj wrote:
>> On 2/9/2012 5:58 PM, Arved Sandstrom wrote:
>>> On 12-02-09 12:15 PM, BGB wrote:
>>>> On 2/9/2012 3:24 AM, Arved Sandstrom wrote:
>>> [ SNIP ]
>>>>
>>>>> Consider line-oriented files/messages like .properties files: these
>>>>> can
>>>>> describe hierarchical structures perfectly well if you've got an
>>>>> understood key=value syntax, specifically with a hierarchy-supporting
>>>>> syntax for the keys. Easy to read and edit, easy to parse.
>>>>
>>>> yes, but this defeats your own prior point, namely indirectly asserting
>>>> that line-oriented == flat-structure.
>>>
>>> Minor quibble, I didn't make such a point, not even indirectly. You may
>>> be confusing me with Arne.
>>>
>>>> point is, one can have hierarchical line-oriented files.
>>> [ SNIP ]
>>>
>>> Yes.
>>
>> You can have non flat structures other than XML, but parsing
>> quickly becomes very complex.
>>
>
> IMO: not particularly...
>
>
> tree-structured line-oriented text formats are not that much more
> complicated to parse than flat-list line-oriented text formats (the only
> obvious difference is that rather than a single loop, one typically has
> multiple loops).
>
> it all still basically amounts to:
> read a line;
> split the string;
> do something with the split strings.

????

But that not how you work with XML.

You load it into a DOM and use XPath to pull
out what you need.

> personally, I like recursive hand-written descent parsing, as it is
> fairly straightforwards and doesn't depend on external tools.

You need to depend on some tools.

If you use Java then you depend on standard Java library.

Standard Java library contains several XML parsers.

No extra dependency for Java.

Arne

[toc] | [prev] | [next] | [standalone]


#11943

FromLew <lewbloch@gmail.com>
Date2012-02-11 12:06 -0800
Message-ID<9002427.542.1328990777214.JavaMail.geo-discussion-forums@pbcpd10>
In reply to#11937
Arne Vajhøj wrote:
> BGB wrote:
>> it all still basically amounts to:
>> read a line;
>> split the string;
>> do something with the split strings.
> 
> ????
> 
> But that not how you work with XML.
> 
> You load it into a DOM and use XPath to pull
> out what you need.

Or you scan it with SAX or StAX and deal with XML-parsing events, or 
you run the schema through JAXB and let it generate all the parsing 
classes for you, or you use one of the many other standard libraries 
in Java (or if not Java, in your favorite platform) that are available 
for free.

>> personally, I like recursive hand-written descent parsing, as it is
>> fairly straightforwards and doesn't depend on external tools.

The number-theorist can never pass a calculus exam because he's only 
just reinvented calculus by the end of the hour.

> You need to depend on some tools.
> 
> If you use Java then you depend on standard Java library.
> 
> Standard Java library contains several XML parsers.
> 
> No extra dependency for Java.

The thing is, BGB, that your macho programming style is impractical 
and not very justifiable. It's wonderful that you have reinvented the 
programming world and all, and are so clever and knowledgeable, but 
most of us programmers work in a workaday pragmatic environment where 
best practices really do save the day. That means use the standards, 
and the abundant tools that support them, and give up our egos that 
make us feel that superman heroics are the only available path. I 
sincerely hope that readers of this thread can understand the manifest 
shortcomings of the approaches that you've espoused here. (And that 
they can disentangle themselves from your fetching but irrelevant 
analogies.) To get the job done, to get it done right, and to minimize both error and development time, use the standards.

XML is just fine for just about every purpose to which it's put. 
That's why it's popular now. People who cavil about "bandwidth" and 
"10 Hz network messages" are tossing us red-herring sashimi. You 
aren't going to get 10+ Hz message exchanges over the WAN. For 
realistic message rates, XML suits beautifully. I speak from 
experience with many, many projects that used every conceivable 
message format from binary to CSV to custom to XML to protocol buffers 
to JSON, and XML has distinct advantages. Its purported disadvantages 
of bulk and bandwidth turn out to be non-issues in practice. Really. 
That's real.

So, dear future readers, stick with what's known to be true by people 
who actually do this work, not by some armchair theorist in a darkened 
room who thinks that he has to do everything by hand and wants the 
rest of us to follow his suboptimal strategy.

-- 
Lew

[toc] | [prev] | [next] | [standalone]


#11945

FromArne Vajhøj <arne@vajhoej.dk>
Date2012-02-11 15:18 -0500
Message-ID<4f36cd1e$0$289$14726298@news.sunsite.dk>
In reply to#11943
On 2/11/2012 3:06 PM, Lew wrote:
> Arne Vajhøj wrote:
>> BGB wrote:
>>> it all still basically amounts to:
>>> read a line;
>>> split the string;
>>> do something with the split strings.
>>
>> ????
>>
>> But that not how you work with XML.
>>
>> You load it into a DOM and use XPath to pull
>> out what you need.
>
> Or you scan it with SAX or StAX and deal with XML-parsing events, or
> you run the schema through JAXB and let it generate all the parsing
> classes for you, or you use one of the many other standard libraries
> in Java (or if not Java, in your favorite platform) that are available
> for free.

If the size of the XML file allows it then DOM and XPath is
usually the least and most readable code.

>> You need to depend on some tools.
>>
>> If you use Java then you depend on standard Java library.
>>
>> Standard Java library contains several XML parsers.
>>
>> No extra dependency for Java.
>
> The thing is, BGB, that your macho programming style is impractical
> and not very justifiable. It's wonderful that you have reinvented the
> programming world and all, and are so clever and knowledgeable, but
> most of us programmers work in a workaday pragmatic environment where
> best practices really do save the day. That means use the standards,
> and the abundant tools that support them, and give up our egos that
> make us feel that superman heroics are the only available path. I
> sincerely hope that readers of this thread can understand the manifest
> shortcomings of the approaches that you've espoused here. (And that
> they can disentangle themselves from your fetching but irrelevant
> analogies.) To get the job done, to get it done right, and to minimize both error and development time, use the standards.

Bad day??

> XML is just fine for just about every purpose to which it's put.
> That's why it's popular now. People who cavil about "bandwidth" and
> "10 Hz network messages" are tossing us red-herring sashimi. You
> aren't going to get 10+ Hz message exchanges over the WAN. For
> realistic message rates, XML suits beautifully. I speak from
> experience with many, many projects that used every conceivable
> message format from binary to CSV to custom to XML to protocol buffers
> to JSON, and XML has distinct advantages. Its purported disadvantages
> of bulk and bandwidth turn out to be non-issues in practice. Really.
> That's real.

????

The industry standard for smartphone apps and AJAX web apps
are JSON because the bandwidth actually matters.

Arne

[toc] | [prev] | [next] | [standalone]


#11963

FromBGB <cr88192@hotmail.com>
Date2012-02-11 23:03 -0700
Message-ID<jh7kp3$30u$1@news.albasani.net>
In reply to#11945
On 2/11/2012 1:18 PM, Arne Vajhøj wrote:
> On 2/11/2012 3:06 PM, Lew wrote:
>> Arne Vajhøj wrote:
>>> BGB wrote:
>>>> it all still basically amounts to:
>>>> read a line;
>>>> split the string;
>>>> do something with the split strings.
>>>
>>> ????
>>>
>>> But that not how you work with XML.
>>>
>>> You load it into a DOM and use XPath to pull
>>> out what you need.
>>
>> Or you scan it with SAX or StAX and deal with XML-parsing events, or
>> you run the schema through JAXB and let it generate all the parsing
>> classes for you, or you use one of the many other standard libraries
>> in Java (or if not Java, in your favorite platform) that are available
>> for free.
>
> If the size of the XML file allows it then DOM and XPath is
> usually the least and most readable code.
>
>>> You need to depend on some tools.
>>>
>>> If you use Java then you depend on standard Java library.
>>>
>>> Standard Java library contains several XML parsers.
>>>
>>> No extra dependency for Java.
>>
>> The thing is, BGB, that your macho programming style is impractical
>> and not very justifiable. It's wonderful that you have reinvented the
>> programming world and all, and are so clever and knowledgeable, but
>> most of us programmers work in a workaday pragmatic environment where
>> best practices really do save the day. That means use the standards,
>> and the abundant tools that support them, and give up our egos that
>> make us feel that superman heroics are the only available path. I
>> sincerely hope that readers of this thread can understand the manifest
>> shortcomings of the approaches that you've espoused here. (And that
>> they can disentangle themselves from your fetching but irrelevant
>> analogies.) To get the job done, to get it done right, and to minimize
>> both error and development time, use the standards.
>
> Bad day??
>
>> XML is just fine for just about every purpose to which it's put.
>> That's why it's popular now. People who cavil about "bandwidth" and
>> "10 Hz network messages" are tossing us red-herring sashimi. You
>> aren't going to get 10+ Hz message exchanges over the WAN. For
>> realistic message rates, XML suits beautifully. I speak from
>> experience with many, many projects that used every conceivable
>> message format from binary to CSV to custom to XML to protocol buffers
>> to JSON, and XML has distinct advantages. Its purported disadvantages
>> of bulk and bandwidth turn out to be non-issues in practice. Really.
>> That's real.
>
> ????
>
> The industry standard for smartphone apps and AJAX web apps
> are JSON because the bandwidth actually matters.
>

yeah.


and, in fact 10Hz asynchronous throughput is possible... (this is 
typically how most online gaming works).

one can't reasonably get 10Hz synchronous (IOW: request/response), due 
to ping times, but this is a different matter.


I am not claiming one can do request/response at 10Hz, as this probably 
would be impossible over a WAN (at reasonable ping times).

online gaming generally isn't based on request/response though, so the 
problems of not being able to get responses at this rate don't really 
matter too much (the user may still notice the results of bad ping 
times, namely stuff being out of place or seemingly teleporting around 
or similar, but this is a different issue).


TCP has a throughput limit due to ping, where assuming unlimited 
bandwidth but a 200ms ping, one will still be limited to somewhere 
around 320kB/s or so (assuming a 64kB window).

however, as is, there are typically bandwidth constraints as well (but, 
they are a little more subject to fudging).

the main goal is mostly to send everything at a bandwidth low enough 
that the connection doesn't risk getting backlogged (depends on internet 
connection, but 16-32 kB/s seems reasonably safe, as internet radio 
often tends to operate in this range, say, 192kbps is 24kB/s).

stalls are also a potential risk as well.


or such...

[toc] | [prev] | [next] | [standalone]


#11975

FromArne Vajhøj <arne@vajhoej.dk>
Date2012-02-12 09:27 -0500
Message-ID<4f37cc42$0$281$14726298@news.sunsite.dk>
In reply to#11963
On 2/12/2012 1:03 AM, BGB wrote:
> On 2/11/2012 1:18 PM, Arne Vajhøj wrote:
>> On 2/11/2012 3:06 PM, Lew wrote:
>>> XML is just fine for just about every purpose to which it's put.
>>> That's why it's popular now. People who cavil about "bandwidth" and
>>> "10 Hz network messages" are tossing us red-herring sashimi. You
>>> aren't going to get 10+ Hz message exchanges over the WAN. For
>>> realistic message rates, XML suits beautifully. I speak from
>>> experience with many, many projects that used every conceivable
>>> message format from binary to CSV to custom to XML to protocol buffers
>>> to JSON, and XML has distinct advantages. Its purported disadvantages
>>> of bulk and bandwidth turn out to be non-issues in practice. Really.
>>> That's real.
>>
>> ????
>>
>> The industry standard for smartphone apps and AJAX web apps
>> are JSON because the bandwidth actually matters.
>
> yeah.

But please note that you do not invent your own JSON parser
either - you use something already done. In Java there are json.org,
gson etc..

> and, in fact 10Hz asynchronous throughput is possible... (this is
> typically how most online gaming works).
>
> one can't reasonably get 10Hz synchronous (IOW: request/response), due
> to ping times, but this is a different matter.
>
> I am not claiming one can do request/response at 10Hz, as this probably
> would be impossible over a WAN (at reasonable ping times).
>
> online gaming generally isn't based on request/response though, so the
> problems of not being able to get responses at this rate don't really
> matter too much (the user may still notice the results of bad ping
> times, namely stuff being out of place or seemingly teleporting around
> or similar, but this is a different issue).
>
> TCP has a throughput limit due to ping, where assuming unlimited
> bandwidth but a 200ms ping, one will still be limited to somewhere
> around 320kB/s or so (assuming a 64kB window).
>
> however, as is, there are typically bandwidth constraints as well (but,
> they are a little more subject to fudging).
>
> the main goal is mostly to send everything at a bandwidth low enough
> that the connection doesn't risk getting backlogged (depends on internet
> connection, but 16-32 kB/s seems reasonably safe, as internet radio
> often tends to operate in this range, say, 192kbps is 24kB/s).
>
> stalls are also a potential risk as well.

I can not follow you way of thinking. With multiple interactions
in parallel there are no strict correlation between latency and
throughput.

Arne

[toc] | [prev] | [next] | [standalone]


#12000

FromBGB <cr88192@hotmail.com>
Date2012-02-12 13:33 -0700
Message-ID<jh97nu$6ao$1@news.albasani.net>
In reply to#11975
On 2/12/2012 7:27 AM, Arne Vajhøj wrote:
> On 2/12/2012 1:03 AM, BGB wrote:
>> On 2/11/2012 1:18 PM, Arne Vajhøj wrote:
>>> On 2/11/2012 3:06 PM, Lew wrote:
>>>> XML is just fine for just about every purpose to which it's put.
>>>> That's why it's popular now. People who cavil about "bandwidth" and
>>>> "10 Hz network messages" are tossing us red-herring sashimi. You
>>>> aren't going to get 10+ Hz message exchanges over the WAN. For
>>>> realistic message rates, XML suits beautifully. I speak from
>>>> experience with many, many projects that used every conceivable
>>>> message format from binary to CSV to custom to XML to protocol buffers
>>>> to JSON, and XML has distinct advantages. Its purported disadvantages
>>>> of bulk and bandwidth turn out to be non-issues in practice. Really.
>>>> That's real.
>>>
>>> ????
>>>
>>> The industry standard for smartphone apps and AJAX web apps
>>> are JSON because the bandwidth actually matters.
>>
>> yeah.
>
> But please note that you do not invent your own JSON parser
> either - you use something already done. In Java there are json.org,
> gson etc..
>

in Java, yes, one may use the libraries.

on the C end, one may choose to throw one together, or use a JavaScript 
VM if one is available, ...

it all depends.


>> and, in fact 10Hz asynchronous throughput is possible... (this is
>> typically how most online gaming works).
>>
>> one can't reasonably get 10Hz synchronous (IOW: request/response), due
>> to ping times, but this is a different matter.
>>
>> I am not claiming one can do request/response at 10Hz, as this probably
>> would be impossible over a WAN (at reasonable ping times).
>>
>> online gaming generally isn't based on request/response though, so the
>> problems of not being able to get responses at this rate don't really
>> matter too much (the user may still notice the results of bad ping
>> times, namely stuff being out of place or seemingly teleporting around
>> or similar, but this is a different issue).
>>
>> TCP has a throughput limit due to ping, where assuming unlimited
>> bandwidth but a 200ms ping, one will still be limited to somewhere
>> around 320kB/s or so (assuming a 64kB window).
>>
>> however, as is, there are typically bandwidth constraints as well (but,
>> they are a little more subject to fudging).
>>
>> the main goal is mostly to send everything at a bandwidth low enough
>> that the connection doesn't risk getting backlogged (depends on internet
>> connection, but 16-32 kB/s seems reasonably safe, as internet radio
>> often tends to operate in this range, say, 192kbps is 24kB/s).
>>
>> stalls are also a potential risk as well.
>
> I can not follow you way of thinking. With multiple interactions
> in parallel there are no strict correlation between latency and
> throughput.
>

there is a rough correlation though.


for the part about TCP, this was related to how TCP worked (in its 
traditional form), namely the existence of a 64kB maximum window size.

apparently, this is out of date, as there is a feature known as 
RFC-1323, which is enabled by default on Windows Vista and newer, which 
allows a larger TCP window.

http://tools.ietf.org/html/rfc1323


for the part about moderating kB/s, this has a lot more to do with a 
users' internet connection.

say, hypothetically, a user has dial-up.

now, what if the data being sent does not fit over dial-up (one is 
trying to send 10kB/s, but a 56k modem can only handle ~6.5kB/s or so)? 
well, then, the connection will backlog (the connection will send at the 
rate it can send, and anything else will have to wait).

similar limits may exist over the internet, but in a less direct form:
consider, the internet is prone to occasionally drop a packet here or there.

so, stream is going over the internet, and a (single) packet drops, what 
happens:
well, all the data up to the dropped packet reaches the other end, the 
other end may send a packet back indicating the point recieved;
the sender will start resending data from that point;
the reciever will start transmitting again.

this results in essentially a ping-time delay in which no data can be sent.

if the sender is sending messages at a fixed rate, what happens?
well then, the messages will pile up, waiting to be sent;
after transmission resumes, several updates worth of data need to be sent;
if all of the updates fit within the bandwidth of the connection 
(end-to-end), then there is may be no obvious stall (updates can all be 
sent at full speed);
if the enough data back-logs so as to exceed the bandwidth available, 
then it has to wait to be sent, and if the sender just keeps naively 
sending updates, then essentially one gets a stall (and the data being 
received by the receiver will start becoming progressively more 
out-of-date).


these properties can be observed with things like internet radio and 
video streaming (if the connection is fast enough, playback happens in 
real-time without obvious stalls or re-buffering, even though the rate 
at which the data comes over the internet is often very irregular).

similar also applies to internet telephony as well.


if one tries to operate within a fixed-bandwidth window, similar to 
internet radio, most minor stalls can be glossed over (this limit being 
a bit lower than the end-to-end transfer rate of the connection). going 
lower is better, since the lower one goes, the more room there for error 
there is.

the main issue is, namely, that the data being sent has to be able to 
fit within these bandwidth limits (hence, why data compression is highly 
desirable in this case).


an online game basically amounts to a bidirectional stream between the 
client and server, with the server sending out a stream of updates 
(typically, everything going on in the immediate view of the client), 
and the client sends a stream of their attempted actions (in response to 
what they see on screen).

if everything is working well, then the delays and irregularities of 
their internet connection is mostly hidden, and to them it all seems 
like they are interacting with the world in real-time (usually there is 
a lot of trickery here as well, mostly based around linear extrapolation 
and so on).

side note: each end may transmit time-stamps as part of their updates, 
and the other end may transmit the last-received timestamps, partly so 
that the timing delays can be estimated and partially compensated for.


another (similar concept) for players playing games is the concept of 
"leading", where a person will take aim at a moving enemy, estimate the 
speed of the projectile and where the enemy will be at the time, and aim 
and fire at that location instead (then the enemy will essentially "run 
into" the traveling projectile). note that if a player always aims at 
where the enemy is "right now", very often they will miss (as by the 
time the projectile reaches the destination, the enemy has already moved 
out of the way).

so, the game does similar in an attempt to hide the "travel time" that 
is the internet.


or such...

[toc] | [prev] | [next] | [standalone]


Page 1 of 4  [1] 2 3 4  Next page →

Back to top | Article view | comp.lang.java.programmer


csiph-web