Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.java.programmer > #29891 > unrolled thread

java unix domain sockets ... or substitute

Back to article view | Back to comp.lang.java.programmer

Contents

  java unix domain sockets ... or substitute Andreas Leitgeb <avl@auth.logic.tuwien.ac.at> - 2014-09-03 17:26 +0000
    Re: java unix domain sockets ... or substitute Wayne <nospam@all.invalid> - 2014-09-03 13:36 -0400
      Re: java unix domain sockets ... or substitute Fredrik Jonson <fredrik@jonson.org> - 2014-09-04 07:42 +0000
      Re: java unix domain sockets ... or substitute Andreas Leitgeb <avl@auth.logic.tuwien.ac.at> - 2014-09-04 07:52 +0000
    Re: java unix domain sockets ... or substitute Josip Almasi <joe@vrspace.org> - 2014-09-04 09:45 +0200
      Re: java unix domain sockets ... or substitute Andreas Leitgeb <avl@auth.logic.tuwien.ac.at> - 2014-09-04 09:14 +0000
        Re: java unix domain sockets ... or substitute Josip Almasi <joe@vrspace.org> - 2014-09-05 10:20 +0200
    Re: java unix domain sockets ... or substitute Fredrik Jonson <fredrik@jonson.org> - 2014-09-04 08:34 +0000
      Re: java unix domain sockets ... or substitute Andreas Leitgeb <avl@auth.logic.tuwien.ac.at> - 2014-09-05 01:06 +0000
    Re: java unix domain sockets ... or substitute Nigel Wade <nmw@ion.le.ac.uk> - 2014-09-04 11:35 +0100
    Re: java unix domain sockets ... or substitute "Chris Uppal" <chris.uppal@metagnostic.REMOVE-THIS.org> - 2014-09-04 15:11 +0100
      Re: java unix domain sockets ... or substitute Nigel Wade <nmw@ion.le.ac.uk> - 2014-09-04 15:57 +0100
        Re: java unix domain sockets ... or substitute "Chris Uppal" <chris.uppal@metagnostic.REMOVE-THIS.org> - 2014-09-04 16:21 +0100
          Re: java unix domain sockets ... or substitute Andreas Leitgeb <avl@auth.logic.tuwien.ac.at> - 2014-09-05 10:55 +0000
    Re: java unix domain sockets ... or substitute (SOLVED) Andreas Leitgeb <avl@auth.logic.tuwien.ac.at> - 2014-09-05 00:53 +0000
      Re: java unix domain sockets ... or substitute (SOLVED) Josip Almasi <joe@vrspace.org> - 2014-09-05 10:23 +0200

#29891 — java unix domain sockets ... or substitute

I've googled for the terms java unix domain sockets, and it seems like 
this cannot be done with plain Java (and its JSL) alone.

There are a few 3rd party libs (junixsocket & JUDS) which unfortunately
both need native libs, which is a non-starter for me.

My current approach is to use named pipes intead, where the server
blockingly reads on a "main" pipe (opened just like a plain file).

Clients first create a new pair of named pipes, and pass the names to
the server, who in response creates a new thread to communicate over
those pipes and serve client's task that way.

This is somewhat awkward (especially cleaning up of stale named pipes),
and named pipes do not prohibit two readers erroneously on the same pipe.

I wonder, if there is:

 - a new way to do unix domain sockets in Java (that google doesn't yet
    find, or only for non-obvious-to-me keyword).  This also includes
    solutions that involve execution of standard unix tools, just no JNI.

 - a way to negotiate a bidirectional channel between a client and
    the server that doesn't involve creating pipe nodes in the
    filesystem.

The client is non-Java, but otherwise quite flexible.

I don't want to use inet sockets (not even localhost), because not every
user on that machine should be able to connect, and I don't want to stick
any authorization stuff into the protocol.

[toc] | [next] | [standalone]

#29893

On 9/3/2014 1:26 PM, Andreas Leitgeb wrote:
> I've googled for the terms java unix domain sockets, and it seems like 
> this cannot be done with plain Java (and its JSL) alone.
> 
> There are a few 3rd party libs (junixsocket & JUDS) which unfortunately
> both need native libs, which is a non-starter for me.
> 
> My current approach is to use named pipes intead, where the server
> blockingly reads on a "main" pipe (opened just like a plain file).
> 
> Clients first create a new pair of named pipes, and pass the names to
> the server, who in response creates a new thread to communicate over
> those pipes and serve client's task that way.
> 
> This is somewhat awkward (especially cleaning up of stale named pipes),
> and named pipes do not prohibit two readers erroneously on the same pipe.
> 
> I wonder, if there is:
> 
>  - a new way to do unix domain sockets in Java (that google doesn't yet
>     find, or only for non-obvious-to-me keyword).  This also includes
>     solutions that involve execution of standard unix tools, just no JNI.
> 
>  - a way to negotiate a bidirectional channel between a client and
>     the server that doesn't involve creating pipe nodes in the
>     filesystem.
> 
> The client is non-Java, but otherwise quite flexible.
> 
> I don't want to use inet sockets (not even localhost), because not every
> user on that machine should be able to connect, and I don't want to stick
> any authorization stuff into the protocol.
> 

Maybe you can create a Java proxy that the non-Java client can use.
Then you can use Java messaging (Java EE).

A simpler solution may be to use ZeroMQ: <http://zeromq.org/bindings:java>

-- 
Wayne

[toc] | [prev] | [next] | [standalone]

#29899

Wayne wrote:
>  On 9/3/2014 1:26 PM, Andreas Leitgeb wrote:
>
> > I wonder, if there is [...] a way to negotiate a bidirectional channel
> > between a client and the server that doesn't involve creating pipe nodes
> > in the filesystem.
> > 
> > The client is non-Java, but otherwise quite flexible.
> > 
> > I don't want to use inet sockets (not even localhost), because not every
> > user on that machine should be able to connect, and I don't want to
> > stick any authorization stuff into the protocol.
> 
>  A simpler solution may be to use ZeroMQ:
>  <http://zeromq.org/bindings:java>

Zeromq does not solve the OP:s original problem. Yes, Zeromq provides
non-socket based in process communication in the native (c language)
implementation. But if you want to use Zeromq to communicate between
separate processes you need to add connectors that use ordinary internet
sockets or domain sockets.

Note that the pure java implementation of ZeroMQ even routes in process
communication (ipc://) over tcp sockets internally.

https://github.com/zeromq/jeromq

-- 
Fredrik Jonson

[toc] | [prev] | [next] | [standalone]

#29901

Wayne <nospam@all.invalid> wrote:
> On 9/3/2014 1:26 PM, Andreas Leitgeb wrote:
>> I've googled for the terms java unix domain sockets, and it seems like 
>> this cannot be done with plain Java (and its JSL) alone.
>> 
>> There are a few 3rd party libs (junixsocket & JUDS) which unfortunately
>> both need native libs, which is a non-starter for me.
>> 
>> My current approach is to use named pipes intead, where the server
>> blockingly reads on a "main" pipe (opened just like a plain file).
>> 
>> Clients first create a new pair of named pipes, and pass the names to
>> the server, who in response creates a new thread to communicate over
>> those pipes and serve client's task that way.
>> 
>> This is somewhat awkward (especially cleaning up of stale named pipes),
>> and named pipes do not prohibit two readers erroneously on the same pipe.
>> 
>> I wonder, if there is:
>> 
>>  - a new way to do unix domain sockets in Java (that google doesn't yet
>>     find, or only for non-obvious-to-me keyword).  This also includes
>>     solutions that involve execution of standard unix tools, just no JNI.
>> 
>>  - a way to negotiate a bidirectional channel between a client and
>>     the server that doesn't involve creating pipe nodes in the
>>     filesystem.
>> 
>> The client is non-Java, but otherwise quite flexible.
>> 
>> I don't want to use inet sockets (not even localhost), because not every
>> user on that machine should be able to connect, and I don't want to stick
>> any authorization stuff into the protocol.
>> 
>
> Maybe you can create a Java proxy that the non-Java client can use.
> Then you can use Java messaging (Java EE).
>
> A simpler solution may be to use ZeroMQ: <http://zeromq.org/bindings:java>

Unfortunately, this fails on one of my requirements:
 - the Java-binding requires a native lib (thus JNI)

Not sure about the "Java proxy". If the proxy had a solution for IPC,
(unix domain sockets or substitute) then I'd use that solution directly
on the server...

[toc] | [prev] | [next] | [standalone]

#29900

On 09/03/2014 07:26 PM, Andreas Leitgeb wrote:
> I've googled for the terms java unix domain sockets, and it seems like
> this cannot be done with plain Java (and its JSL) alone.
>
> There are a few 3rd party libs (junixsocket & JUDS) which unfortunately
> both need native libs, which is a non-starter for me.
>
> My current approach is to use named pipes intead, where the server
> blockingly reads on a "main" pipe (opened just like a plain file).
>
> Clients first create a new pair of named pipes, and pass the names to
> the server, who in response creates a new thread to communicate over
> those pipes and serve client's task that way.
>
> This is somewhat awkward (especially cleaning up of stale named pipes),
> and named pipes do not prohibit two readers erroneously on the same pipe.
>
> I wonder, if there is:
>
>   - a new way to do unix domain sockets in Java (that google doesn't yet
>      find, or only for non-obvious-to-me keyword).  This also includes
>      solutions that involve execution of standard unix tools, just no JNI.
>
>   - a way to negotiate a bidirectional channel between a client and
>      the server that doesn't involve creating pipe nodes in the
>      filesystem.
>
> The client is non-Java, but otherwise quite flexible.
>
> I don't want to use inet sockets (not even localhost), because not every
> user on that machine should be able to connect, and I don't want to stick
> any authorization stuff into the protocol.

IMHO domain sockets are not in java for good reason - they add little 
value over pipes when used locally, but add complexity, as it's easier 
to work with files/pipes than sockets.
Also, pipes and sockets use the same auth mechanism - filesystem 
permissions.
With domain sockets you'd also get filesystem nodes, just not two for 
each client, with less bookkeeping of stale nodes.
Then again, you also can create pipes in /tmp, and leave cleanup to system.

Now, if you want to utilize local auth, simply put your client 
executable in a specific group, set permissions to dissallow anyone 
outside that group to run it, and add only users allowed into that group.
Then it's all the same which IPC mechanism you use.

But look: either you use local IPC and auth, or you use remote IPC and 
auth mechanisms.
In my experience, it's always better to start with remote IPC used locally.
Like, maybe you host becomes cluster, then your server and client become 
cluster services, with no local IPC.

Regards...

[toc] | [prev] | [next] | [standalone]

#29903

Josip Almasi <joe@vrspace.org> wrote:
> On 09/03/2014 07:26 PM, Andreas Leitgeb wrote:
>> My current approach is to use named pipes intead, where the server
>> blockingly reads on a "main" pipe (opened just like a plain file).
>>
>> Clients first create a new pair of named pipes, and pass the names to
>> the server, who in response creates a new thread to communicate over
>> those pipes and serve client's task that way.
>>
>> This is somewhat awkward (especially cleaning up of stale named pipes),
>> and named pipes do not prohibit two readers erroneously on the same pipe.
>>
>> I wonder, if there is:
>>
>>   - a new way to do unix domain sockets in Java (that google doesn't yet
>>      find, or only for non-obvious-to-me keyword).  This also includes
>>      solutions that involve execution of standard unix tools, just no JNI.
>>
>>   - a way to negotiate a bidirectional channel between a client and
>>      the server that doesn't involve creating pipe nodes in the
>>      filesystem.
>>
>> The client is non-Java, but otherwise quite flexible.
>>
>> I don't want to use inet sockets (not even localhost), because not every
>> user on that machine should be able to connect, and I don't want to stick
>> any authorization stuff into the protocol.
>
> IMHO domain sockets are not in java for good reason - they add little 
> value over pipes when used locally, but add complexity, as it's easier 
> to work with files/pipes than sockets.

Exactly that added complexity is what I now have to mimic with pipes and
where it ends up even more complicated than what I'd think domain sockets
would have been.
As a bonus, I'd be able to detect stale sockets, in case the server crashed,
but I cannot reliably detect a stale named pipe. Accessing it will merely 
block until someone else restarts the server.

> With domain sockets you'd also get filesystem nodes, just not two for 
> each client, with less bookkeeping of stale nodes.
> Then again, you also can create pipes in /tmp, and leave cleanup to system.

The mean-time between reboots is rather long (think of years), so I couldn't
just idly fill /tmp. The rate of client-connects is expected to be high
enough that this would cause problems.

> Now, if you want to utilize local auth, simply put your client 
> executable in a specific group, set permissions to dissallow anyone 
> outside that group to run it, and add only users allowed into that group.

or, as I'm doing it, I set up that nod with "0600" perms for the user.

> In my experience, it's always better to start with remote IPC used locally.
> Like, maybe you host becomes cluster, then your server and client become 
> cluster services, with no local IPC.

This server isn't meant for any kind of "public" access.  The server merely
hosts more or less short-lived apps, that may eventually alter local files or
spawn processes, and each of that "as the user" who started the particular server
instance. Therefore, no one else than the user ought to be able to use the server.
Essentially, the user could just as well start his own java vm to run his app,
bypassing the server process, but then he suffers from java vm startup times.

In a later stage, I'll make it a public(still localhost) server, but then
it'll delegate all the delicate stuff (file access, etc) back to the client...

[toc] | [prev] | [next] | [standalone]

#29917

On 09/04/2014 11:14 AM, Andreas Leitgeb wrote:
> Josip Almasi <joe@vrspace.org> wrote:
>> IMHO domain sockets are not in java for good reason - they add little
>> value over pipes when used locally, but add complexity, as it's easier
>> to work with files/pipes than sockets.
>
> Exactly that added complexity is what I now have to mimic with pipes and
> where it ends up even more complicated than what I'd think domain sockets
> would have been.
> As a bonus, I'd be able to detect stale sockets, in case the server crashed,
> but I cannot reliably detect a stale named pipe. Accessing it will merely
> block until someone else restarts the server.

I think you can detect stale pipes, with fuser and/or lsof - they will 
not be opened by any processes.
Something like
for FILE in `find /tmp -type p`
do
   PROCCNT=`fuser $FILE | wc -w`
   if (( $PROCCNT == 1 ))
     fuser -k $FILE
   elif (( $PROCCNT == 0 ))
   then
     rm -f $FILE
   fi
done
every (couple of) minute(s).

>> With domain sockets you'd also get filesystem nodes, just not two for
>> each client, with less bookkeeping of stale nodes.
>> Then again, you also can create pipes in /tmp, and leave cleanup to system.
>
> The mean-time between reboots is rather long (think of years), so I couldn't
> just idly fill /tmp. The rate of client-connects is expected to be high
> enough that this would cause problems.

Cleaning /tmp is usually night-time cron job.
Think you only need one or two cron jobs to keep your pipes under control.
I haven't seen AIX, Solaris nor ksh in years though.

Regards...

[toc] | [prev] | [next] | [standalone]

#29902

Andreas Leitgeb wrote:

>  I've googled for the terms java unix domain sockets, and it seems like
>  this cannot be done with plain Java (and its JSL) alone.

That's my understanding too, unfortunately. I haven't found any unix socket
library for Java that does not use JNI.

>  [I need] a way to negotiate a bidirectional channel between a client and
>  the server that doesn't involve creating pipe nodes in the filesystem.
> 
>  I don't want to use inet sockets (not even localhost), because not every
>  user on that machine should be able to connect, and I don't want to stick
>  any authorization stuff into the protocol.

You might want to widen your search to include "shared memory" too. Though
I'm not sure that it solves your original problem.

If platform independence is essential the easiest solution is internet
sockets. You would indeed have to add authorization but that's mostly a
solved problem. If you use Mina or Netty, or some other higher level
library, you can easily find ready made solutions for authorization and
session management.

-- 
Fredrik Jonson

[toc] | [prev] | [next] | [standalone]

#29916

Fredrik Jonson <fredrik@jonson.org> wrote:
> Andreas Leitgeb wrote:
>>  [I need] a way to negotiate a bidirectional channel between a client and
>>  the server that doesn't involve creating pipe nodes in the filesystem.
>
> You might want to widen your search to include "shared memory" too. Though
> I'm not sure that it solves your original problem.

Good point.  With shared memory and semaphores I probably could have
done something.  Not sure, though, if that would have been less complex
than the substitute solution with named pipes that I already have.

For other communication needs/profiles it could have been better.

Thanks for bringing it up.

[toc] | [prev] | [next] | [standalone]

#29904

On 03/09/14 18:26, Andreas Leitgeb wrote:
> I've googled for the terms java unix domain sockets, and it seems like
> this cannot be done with plain Java (and its JSL) alone.
>
> There are a few 3rd party libs (junixsocket & JUDS) which unfortunately
> both need native libs, which is a non-starter for me.
>
> My current approach is to use named pipes intead, where the server
> blockingly reads on a "main" pipe (opened just like a plain file).
>
> Clients first create a new pair of named pipes, and pass the names to
> the server, who in response creates a new thread to communicate over
> those pipes and serve client's task that way.
>
> This is somewhat awkward (especially cleaning up of stale named pipes),
> and named pipes do not prohibit two readers erroneously on the same pipe.
>
> I wonder, if there is:
>
>   - a new way to do unix domain sockets in Java (that google doesn't yet
>      find, or only for non-obvious-to-me keyword).  This also includes
>      solutions that involve execution of standard unix tools, just no JNI.
>
>   - a way to negotiate a bidirectional channel between a client and
>      the server that doesn't involve creating pipe nodes in the
>      filesystem.
>
> The client is non-Java, but otherwise quite flexible.
>
> I don't want to use inet sockets (not even localhost), because not every
> user on that machine should be able to connect, and I don't want to stick
> any authorization stuff into the protocol.
>


Can't you achieve what you want using simple files and ownership/permissions?

A simple solution might be for the server to listen on a randomly assigned port. When the server has been assigned the 
port it creates a well-known file in the filesystem (in /tmp would be a suitable place), and writes the port number to 
that file. The file should be owned by the server's userid, and only accessible by that user (permissions 0x300). The 
File should be set to deleteOnClose().

When a client wants to connect to the server it attempts to read the port number from the well-known file. If the file 
doesn't exist it can assume the server isn't running. If it can't open the file then it doesn't have permission to talk 
to the server. If it successfully reads the port number it can attempt to connect to the server on that port.

It's still possible for a malicious client to connect to the server by port-scanning. You could feasibly include a 
random "key" in the file along with the port number, and the client needs to know both the port and the key. How much 
security do you need on your system?

[toc] | [prev] | [next] | [standalone]

#29906

Andreas Leitgeb wrote:
> - a way to negotiate a bidirectional channel between a client and
>    the server that doesn't involve creating pipe nodes in the
>    filesystem.

Seems quite obvious that this calls for the server to listen on a local TCP/IP 
port (and probably only accept incoming connections from localhost), then layer 
whatever kind of security you want over that.

If you must use Unix group permissions, then Nigel Wade's suggestion of having 
the server write a random number to a file that only it can write and only they 
can read, and using that for validation, seems adequate.  But note that that is 
vulnerable to network sniffing.  But that may not matter to you.  Even if not, 
I'd be inclined to change the number after each connection attempt (sucessfull 
or not), and have the clients retry in case of failure (race conditions, etc).

A better solution for many (most?) purposes would be to use SSL/TLS instead of 
raw TCP/IP.  The server would need a certificate, but you could use a 
self-signed cert.

If you go that route then you could still use Nigel's idea to validate by 
group, or you could set up your own password system that was submitted/checked 
over SSL.  Alternatively you could use client certificates at the SSL level.

A lot -- in fact everything -- depends on what /kind/ of security you want, and 
how /much/ security you want.

    -- chris 

[toc] | [prev] | [next] | [standalone]

#29909

On 04/09/14 15:11, Chris Uppal wrote:
> If you must use Unix group permissions, then Nigel Wade's suggestion of having
> the server write a random number to a file that only it can write and only they
> can read, and using that for validation, seems adequate.  But note that that is
> vulnerable to network sniffing.

Network sniffing ought not to be much of a problem.

If the server only listens on the loopback interface then external exploitation shouldn't be possible. A normal user 
should not be able to sniff the lookback interface, so the attacker would need root access on the local host. If an 
attacker has root access then all bets are off, and protecting the app. server would be low on the list of priorities.

[toc] | [prev] | [next] | [standalone]

#29911

Nigel Wade wrote:
> On 04/09/14 15:11, Chris Uppal wrote:
> > If you must use Unix group permissions, then Nigel Wade's suggestion of
> > having the server write a random number to a file that only it can write 
> > and
> > only they can read, and using that for validation, seems adequate.  But 
> > note that
> > that is vulnerable to network sniffing.
>
> Network sniffing ought not to be much of a problem.

Agreed that if the server only listens on the local interface it's difficult to 
eavesdrop (and that's true for Windows as well as *nix as far as I know).  But 
it feels rather fragile to me.

"The box that hosts the clients is getting a bit overloaded, can we move some 
processing onto another box somewhere?"

"Sure, we just move the central server process onto <some machine>.  It's 
really just a configuration change.  Should be easy"

...

"oh"


But, of course, it all depends on Andreas's needs.  Even across the LAN, my 
suggestion of changing the "password" after each use might be sufficient (or it 
might be the worst idea /ever/)...

    -- chris 

[toc] | [prev] | [next] | [standalone]

#29921

Chris Uppal <chris.uppal@metagnostic.REMOVE-THIS.org> wrote:
> "The box that hosts the clients is getting a bit overloaded, can we move some 
> processing onto another box somewhere?"
>
> "Sure, we just move the central server process onto <some machine>.  It's 
> really just a configuration change.  Should be easy"
>
> ...
>
> "oh"

If a need for remote-ability arises in future, I'll just
add a new "InetServer" thread to the PipeServer thread.
The InetServer then can still handle any authorization
needs - it'll deal with a different client, anyway.

Judging my situation here leads me to think IAGNI. :-)

[toc] | [prev] | [next] | [standalone]

#29915 — Re: java unix domain sockets ... or substitute (SOLVED)

Andreas Leitgeb <avl@auth.logic.tuwien.ac.at> wrote:
> My current approach is to use named pipes intead, where the server
> blockingly reads on a "main" pipe (opened just like a plain file).
>
> Clients first create a new pair of named pipes, and pass the names to
> the server, who in response creates a new thread to communicate over
> those pipes and serve client's task that way.

I've now settled on this approach.

> This is somewhat awkward (especially cleaning up of stale named pipes),
> and named pipes do not prohibit two readers erroneously on the same pipe.

I found a solution for keeping the number of stale pipes low enough.
The trick is to delete them as soon as they're opened by both sides.
If a process dies abruptly during its pay-work, then the pipes are
already cleaned up.

[toc] | [prev] | [next] | [standalone]

#29918 — Re: java unix domain sockets ... or substitute (SOLVED)

On 09/05/2014 02:53 AM, Andreas Leitgeb wrote:
> Andreas Leitgeb <avl@auth.logic.tuwien.ac.at> wrote:
>> My current approach is to use named pipes intead, where the server
>> blockingly reads on a "main" pipe (opened just like a plain file).
>>
>> Clients first create a new pair of named pipes, and pass the names to
>> the server, who in response creates a new thread to communicate over
>> those pipes and serve client's task that way.
>
> I've now settled on this approach.
>
>> This is somewhat awkward (especially cleaning up of stale named pipes),
>> and named pipes do not prohibit two readers erroneously on the same pipe.
>
> I found a solution for keeping the number of stale pipes low enough.
> The trick is to delete them as soon as they're opened by both sides.
> If a process dies abruptly during its pay-work, then the pipes are
> already cleaned up.

Neat trick!:)

Regards...

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.java.programmer

csiph-web