Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #196898 > unrolled thread

Beowulf gone?

Back to article view | Back to linux.debian.user

Contents

  Beowulf gone? Hans <hans.ullrich@loop.de> - 2018-06-24 21:20 +0200
    Re: Beowulf gone? Richard Owlett <rowlett@cloud85.net> - 2018-06-24 21:40 +0200
    Re: Beowulf gone? Ionel Mugurel Ciobîcă <I.M.Ciobica@upcmail.nl> - 2018-06-24 23:00 +0200
      Re: Beowulf gone? Carl Fink <carlf@panix.com> - 2018-06-24 23:30 +0200
    Re: Beowulf gone? Michael Stone <mstone@debian.org> - 2018-06-25 04:40 +0200
      Re: Beowulf gone? Hans <hans.ullrich@loop.de> - 2018-06-25 08:20 +0200
        Re: Beowulf gone? <tomas@tuxteam.de> - 2018-06-25 09:00 +0200
          Re: Beowulf gone? deloptes <deloptes@gmail.com> - 2018-06-25 22:30 +0200
            Re: Beowulf gone? Michael Stone <mstone@debian.org> - 2018-06-25 23:10 +0200
            Re: Beowulf gone? Mike Castle <dalgoda+debian@gmail.com> - 2018-06-26 07:00 +0200
        Re: Beowulf gone? Mike Castle <dalgoda+debian@gmail.com> - 2018-06-25 10:20 +0200
          Re: Beowulf gone? Hans <hans.ullrich@loop.de> - 2018-06-25 10:40 +0200
            Re: Beowulf gone? Curt <curty@free.fr> - 2018-06-25 14:10 +0200

#196898 — Beowulf gone?

Hi folks,

I am interested in clustering a server for educacation purposes,  Asd far as I 
remembewr, there had bbeen some debian packages called "beowulf" or so, 

However, I may be wrong and saw them in SuSE-linux not in debian.

Today I found no beowqulf packages any more. Question: If Í want to create a 
cluster,  are the "mpich"-packes today the correct choice? (Is mpich the 
successor to beowulf?)

Please apologize, I still did not read much of the beowulf documentations, my 
question is more aimed, if mpich is the preferred thing to take a look at or 
not.

Thanks for any hints.

Best

Hans

[toc] | [next] | [standalone]

#196899

On 06/24/2018 02:17 PM, Hans wrote:
> Hi folks,
> 
> I am interested in clustering a server for educacation purposes,  Asd far as I
> remembewr, there had bbeen some debian packages called "beowulf" or so,
> 
> However, I may be wrong and saw them in SuSE-linux not in debian.
> 
> Today I found no beowqulf packages any more. Question: If Í want to create a
> cluster,  are the "mpich"-packes today the correct choice? (Is mpich the
> successor to beowulf?)
> 
> Please apologize, I still did not read much of the beowulf documentations, my
> question is more aimed, if mpich is the preferred thing to take a look at or
> not.
> 
> Thanks for any hints.
> 

Can't answer your question, but can give useful pointers.
Whatever is being currently used would be part of a Debian Pure Blend.
[https://www.debian.org/blends/]
The education related blend
[https://wiki.debian.org/DebianEdu/Introduction]
There are pointers for related servers at
[https://packages.debian.org/wheezy/education-workstation]

There is also a Blends mailing list.
[https://lists.debian.org/debian-blends/]

HTH

[toc] | [prev] | [next] | [standalone]

#196900

On 24-06-2018, at 21h 17'25", Hans wrote about "Beowulf gone?"
> Hi folks,
> 
> I am interested in clustering a server for educacation purposes,  Asd far as I 
> remembewr, there had bbeen some debian packages called "beowulf" or so, 
> 
> However, I may be wrong and saw them in SuSE-linux not in debian.
> 
> Today I found no beowqulf packages any more. Question: If Í want to create a 
> cluster,  are the "mpich"-packes today the correct choice? (Is mpich the 
> successor to beowulf?)
> 
> Please apologize, I still did not read much of the beowulf documentations, my 
> question is more aimed, if mpich is the preferred thing to take a look at or 
> not.
> 
> Thanks for any hints.
> 
> Best
> 
> Hans
> 

Hans,

mpich is an implementation of MPI (message passing interface), that is
a way of parallelization of a code to run on more than one core, either
on the same host or over few hosts.

beowulf is a way to configure and maintain a PC-cluster.

I hope it is clear now.

Kind regards,
 Ionel

[toc] | [prev] | [next] | [standalone]

#196901

It would seem the Debian Beowulf project petered out almost 20 years 
ago. It was based here:

https://www.debian.org/ports/beowulf/

The last non-spam message to the debian-beowulf list was in 2010 or so.
https://lists.debian.org/debian-beowulf/

-- 
Carl Fink          carl@finknetwork.com
Thinking and logic and stuff at Reasonably Literate
http://reasonablyliterate.com

[toc] | [prev] | [next] | [standalone]

#196907

On Sun, Jun 24, 2018 at 09:17:25PM +0200, Hans wrote:
>I am interested in clustering a server for educacation purposes,  Asd far as I
>remembewr, there had bbeen some debian packages called "beowulf" or so,

beowulf was a system at NASA Goddard 20 years ago. The basic idea of 
clustering computers was around before that, and is still around today.  
There are many different kinds of clusters depending on what you're 
trying to accomplish. Availability? Scaling to massive data sets? Can 
the work be easily distributed without much communication or does one 
computation depend on the results of many other computations? The 
problem determines the software & hardware. As a starting point you 
could look at hadoop and spark and openmpi and openstack to get some 
idea of the scale and breadth of the question. 

Mike Stone

[toc] | [prev] | [next] | [standalone]

#196909

First of all, thanks for making things clearer. So it looks like, that Beowulf 
is dead since many years. 

Is there a successor? I have in my mind to use a cluster for rendering videos 
and thinking, a cluster would speed up things.

At trhe moment I am still collecting ideas.

Best 

Hans

[toc] | [prev] | [next] | [standalone]

#196911

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Mon, Jun 25, 2018 at 08:14:28AM +0200, Hans wrote:
> First of all, thanks for making things clearer. So it looks like, that Beowulf 
> is dead since many years. 
> 
> Is there a successor? I have in my mind to use a cluster for rendering videos 
> and thinking, a cluster would speed up things.

There are *many* successors. Re-read Michael's mail, he even offers
three names.

Most "interesting" things out there are clustered, these days. Google
wouldn't exist without.

The problem will be for you -- what tasks are you thinking about? I.e.
what are the "things" you mention above you want to "speed up"?

You could do worse than reading Wikipedia's entry for computer
clusters [1].

Cheers

[1] https://en.wikipedia.org/wiki/Computer_cluster
- -- tomás
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iEYEARECAAYFAlswkoEACgkQBcgs9XrR2kYhsgCfQ1nbC+bWBkDF8ckQENQgyDUd
dhwAn087lL5YdoHay3cBoWKSbid1qxik
=QaV4
-----END PGP SIGNATURE-----

[toc] | [prev] | [next] | [standalone]

#196926

tomas@tuxteam.de wrote:

> The problem will be for you -- what tasks are you thinking about? I.e.
> what are the "things" you mention above you want to "speed up"?

but he said he wants to render video

[toc] | [prev] | [next] | [standalone]

#196929

On Mon, Jun 25, 2018 at 10:28:01PM +0200, deloptes wrote:
>but he said he wants to render video

And to put that in perspective, one multicore machine with SSE 
instructions is *vastly* more powerful than a 90s-era cluster for that 
specific task, and GPU-based rendering is even more powerful. As someone 
else pointed out, the best approach for this problem is to choose the 
software first, then find out what options it has.

Mike Stone

[toc] | [prev] | [next] | [standalone]

#196938

I took rendering video to be an immediate example, but not necessarily
the only thing of interest.

[toc] | [prev] | [next] | [standalone]

#196913

Down to it's basic, rendering videos is nothing more than a simple
map-reduce, partioning a workload in a bunch of identical bits of
processing.  That could be done with N machines and a few simple shell
scripts.  Not really any need for anything fancy.  What the fancier
software gives you is stuff like automatic retries, fault tolerance,
fancy ways of tracking the process of a job, and so on.  But they
likely require more management to maintain.

Other reasons for a cluster might be web serving.  You want to be able
to handle some amount of queries per second with a reasonable latency
for most of those queries.  That may require you to scale in different
ways.  You need more machines to handle serving up the web content, a
bit more to handle the various types of processing that needs to be
done, some sort of scaling solution for your data store.  Still
parallelism, but each thing happening in parallel is likely a
different type of task.  A simple thing like Puppet or Chef could help
you keep your machines in order.

Maybe you don't need the computational power of a bunch of CPUs, but
you need all of the memory for something?  Maybe you need a bunch of
machines with 128G of ram to look like a single machine with 1P of
ram.  So you need to have something specialized for moving bits around
efficiently (either code to the data, or data to the code).

Maybe you want to have different types of pipelines that do completely
different things, and you want to make efficient use of the machinery
you have.  Say, you're doing both of the above, and during some hours
of the day, you need more processing power to go towards web serving,
you take away resources from you map-reduce.  During off hours, you
can let the MR have more processing power.  Now you're dynamically
trading off resources, and this is where some sort of clustering
management might become useful.

There is not likely to be any one solution for everyone.  So no longer
having anything exactly like Beowulf or some sort of direct
replacement is not suprising.  Likely stuff was learned from Beowulf
and friends.  Some things worked well, some things no, some things
were never used, some things were needed.  Folks take that experience
and build new tools and systems.  Old ones languish.  And no direct
replacement exists.

Likely you need to break down what you really want to do, then look to
see what solutions might work best for you, and experiment with the
various ones, and see which one you like best.

mrc

[toc] | [prev] | [next] | [standalone]

#196914

Am Montag, 25. Juni 2018, 10:13:21 CEST schrieb Mike Castle:
Hi Mike, 

thank you very much for this very informative response. So it looks like, I 
have to search for a solution which will exactly fit my needs. 

The primary goal or interest will be, to render videos (I say some kind of 
"Micro-Pixar-Studio" :)   ), as I was aked for this by some friend.

All responses her in the debian grpup showed me, that things are not as easy 
as I imagined and there is no siomple amnd easy solution.  That is no problem, 
as I am willing to learn and I alrerady found several documentations, whixch I 
will read the next time.

For now, thank you all for yourt hints and informations, and I believe, for 
now this thing can be marked as solved.

Best regards and have always fun!

Hans
 
> Down to it's basic, rendering videos is nothing more than a simple
> map-reduce, partioning a workload in a bunch of identical bits of
> processing.  That could be done with N machines and a few simple shell
> scripts.  Not really any need for anything fancy.  What the fancier
> software gives you is stuff like automatic retries, fault tolerance,
> fancy ways of tracking the process of a job, and so on.  But they
> likely require more management to maintain.
> 
> Other reasons for a cluster might be web serving.  You want to be able
> to handle some amount of queries per second with a reasonable latency
> for most of those queries.  That may require you to scale in different
> ways.  You need more machines to handle serving up the web content, a
> bit more to handle the various types of processing that needs to be
> done, some sort of scaling solution for your data store.  Still
> parallelism, but each thing happening in parallel is likely a
> different type of task.  A simple thing like Puppet or Chef could help
> you keep your machines in order.
> 
> Maybe you don't need the computational power of a bunch of CPUs, but
> you need all of the memory for something?  Maybe you need a bunch of
> machines with 128G of ram to look like a single machine with 1P of
> ram.  So you need to have something specialized for moving bits around
> efficiently (either code to the data, or data to the code).
> 
> Maybe you want to have different types of pipelines that do completely
> different things, and you want to make efficient use of the machinery
> you have.  Say, you're doing both of the above, and during some hours
> of the day, you need more processing power to go towards web serving,
> you take away resources from you map-reduce.  During off hours, you
> can let the MR have more processing power.  Now you're dynamically
> trading off resources, and this is where some sort of clustering
> management might become useful.
> 
> There is not likely to be any one solution for everyone.  So no longer
> having anything exactly like Beowulf or some sort of direct
> replacement is not suprising.  Likely stuff was learned from Beowulf
> and friends.  Some things worked well, some things no, some things
> were never used, some things were needed.  Folks take that experience
> and build new tools and systems.  Old ones languish.  And no direct
> replacement exists.
> 
> Likely you need to break down what you really want to do, then look to
> see what solutions might work best for you, and experiment with the
> various ones, and see which one you like best.
> 
> mrc

[toc] | [prev] | [next] | [standalone]

#196915

On 2018-06-25, Hans <hans.ullrich@loop.de> wrote:
> Am Montag, 25. Juni 2018, 10:13:21 CEST schrieb Mike Castle:
> Hi Mike, 
>
> thank you very much for this very informative response. So it looks like, I 
> have to search for a solution which will exactly fit my needs. 
>
> The primary goal or interest will be, to render videos (I say some kind of 
> "Micro-Pixar-Studio" :)   ), as I was aked for this by some friend.

Cinelerra (can't vouch for it and don't even know whether it's open
source, if you're one of the purists in the crowd) has a render farm
"feature" (Blender, I believe, now includes a network rendering addon).

I suppose (perhaps erroneously) that once you have a working lan set up
the cluster farm (I almost said something else!) shouldn't be too
arduous to configure.

Maybe not up to Pixar snuff, but I would think many video rendering
softs include some kind of distributed rendering feature these days
(although I wouldn't really know, as I ain't a renderer myself).

Bonne chance.

> All responses her in the debian grpup showed me, that things are not as easy 
> as I imagined and there is no siomple amnd easy solution.  That is no problem, 
> as I am willing to learn and I alrerady found several documentations, whixch I 
> will read the next time.
>
> For now, thank you all for yourt hints and informations, and I believe, for 
> now this thing can be marked as solved.
>
> Best regards and have always fun!
>

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.user

csiph-web