Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.java.programmer > #33030

Re: Things have 'moved on' apparently

Show all headers | View raw

On Sun, 08 Nov 2015 18:54:07 +0100, Robert Klemme wrote:

> There are also other cases where it is better to bite the bullet and
> write stuff yourself, even if it is complicated.
>
Been there, done that.

In this case the offending functions were the malloc and B-tree modules 
in the C standard library for DEC's Alpha servers running TrueUNIX.

The root problem was that the OS was Mach-based and the implementation 
seemed to use a single-threaded process to implement each low level clib 
function: at least, we found this problem in several parts of the OS. 
Maybe this is the main reason that Mach message-passing micro-kernel 
architectures have largely disappeared, but I digress...
 
The effect was that, because the btree routine used three chunks of 
memory for each node (IIRC one for key, one for data, one for the three 
pointers), so that lookups (which added unknown nodes to the tree) bottle-
necked on the malloc() process with three calls per node added. We needed 
something like 7,000 lookups/sec but couldn't break 800.

So I rewrote the btree functions to use only one struct per node, which 
gave an immediate speedup to better than 1500 lookups/sec. 

More checks showed that now the sbrk() function that malloc() called each 
time it needed to extend process heap memory had the same problem. So, I 
changed my code to use malloc() to grab MB-sized chunks of heap space and 
and wrote an application level function that allocated btree nodes in my 
chunk of heap and asked for more when it was needed. This gave me 25,000 
looks/sec: job done!

This also showed just how important it is to have enough test data: we 
had 500,000 test data records. If we'd been using the 50 - 200 records 
I've seen a lot of projects get by with this dire performance would have 
been a nasty shock when the system went live instead of showing up at the 
start of integration testing.

> Bottom line: you have to die one death or another
>
If it isn't, it should be.

> there is no general rule - just the usual trade offs.
> 
Spot on.



-- 
martin@   | Martin Gregorie
gregorie. | Essex, UK
org       |

Back to comp.lang.java.programmer | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Things have 'moved on' apparently Eternal Hope <laughingspam99@gmail.com> - 2015-11-06 12:32 +0000
  Re: Things have 'moved on' apparently Kevin McMurtrie <mcmurtrie@pixelmemory.us> - 2015-11-06 08:25 -0800
    Re: Things have 'moved on' apparently Eric Douglas <e.d.programmer@gmail.com> - 2015-11-06 12:02 -0800
      Re: Things have 'moved on' apparently Jerry Stuckle <jstucklex@attglobal.net> - 2015-11-06 19:46 -0500
        Re: Things have 'moved on' apparently Martin Gregorie <martin@address-in-sig.invalid> - 2015-11-07 17:21 +0000
          Re: Things have 'moved on' apparently Jerry Stuckle <jstucklex@attglobal.net> - 2015-11-07 12:50 -0500
        Re: Things have 'moved on' apparently Eric Douglas <e.d.programmer@gmail.com> - 2015-11-09 05:37 -0800
          Re: Things have 'moved on' apparently Jerry Stuckle <jstucklex@attglobal.net> - 2015-11-09 09:00 -0500
          Re: Things have 'moved on' apparently Martin Gregorie <martin@address-in-sig.invalid> - 2015-11-09 14:56 +0000
          Re: Things have 'moved on' apparently Martin Gregorie <martin@address-in-sig.invalid> - 2015-11-09 19:25 +0000
      Re: Things have 'moved on' apparently Arne Vajhøj <arne@vajhoej.dk> - 2015-11-07 21:57 -0500
    Re: Things have 'moved on' apparently Arne Vajhøj <arne@vajhoej.dk> - 2015-11-07 22:02 -0500
      Re: Things have 'moved on' apparently leif.roar@dimnakorr.com - 2015-11-08 02:00 -0600
        Re: Things have 'moved on' apparently Arne Vajhøj <arne@vajhoej.dk> - 2015-11-08 08:44 -0500
          Re: Things have 'moved on' apparently leif.roar@dimnakorr.com - 2015-11-08 13:08 -0600
        Re: Things have 'moved on' apparently Robert Klemme <shortcutter@googlemail.com> - 2015-11-08 18:54 +0100
          Re: Things have 'moved on' apparently Martin Gregorie <martin@address-in-sig.invalid> - 2015-11-08 21:12 +0000
            Re: Things have 'moved on' apparently Arne Vajhøj <arne@vajhoej.dk> - 2015-11-08 17:05 -0500
              Re: Things have 'moved on' apparently Martin Gregorie <martin@address-in-sig.invalid> - 2015-11-08 23:24 +0000
                Re: Things have 'moved on' apparently Arne Vajhøj <arne@vajhoej.dk> - 2015-11-08 19:40 -0500
                Re: Things have 'moved on' apparently Jerry Stuckle <jstucklex@attglobal.net> - 2015-11-08 21:32 -0500
                Re: Things have 'moved on' apparently Martin Gregorie <martin@address-in-sig.invalid> - 2015-11-09 15:15 +0000
      Re: Things have 'moved on' apparently Kevin McMurtrie <mcmurtrie@pixelmemory.us> - 2015-11-08 12:10 -0800
        Re: Things have 'moved on' apparently Jerry Stuckle <jstucklex@attglobal.net> - 2015-11-08 16:00 -0500
        Re: Things have 'moved on' apparently Arne Vajhøj <arne@vajhoej.dk> - 2015-11-08 16:51 -0500
  Re: Things have 'moved on' apparently Jan Burse <janburse@fastmail.fm> - 2015-11-08 00:55 +0100
  Re: Things have 'moved on' apparently Arne Vajhøj <arne@vajhoej.dk> - 2015-11-07 21:28 -0500
    Re: Things have 'moved on' apparently Arne Vajhøj <arne@vajhoej.dk> - 2015-11-07 21:36 -0500
  Re: Things have 'moved on' apparently leif.roar@dimnakorr.com - 2015-11-08 02:12 -0600
  Re: Things have 'moved on' apparently askfor <askfor@nowhere.net> - 2015-11-09 18:02 +0100
  Re: Things have 'moved on' apparently Kevin McMurtrie <mcmurtrie@pixelmemory.us> - 2015-11-10 22:32 -0800
    Re: Things have 'moved on' apparently Eternal Hope <laughingspam99@gmail.com> - 2015-11-11 07:52 +0000
      Re: Things have 'moved on' apparently ross.finlayson@gmail.com - 2015-11-19 23:28 -0800

csiph-web