Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c > #161971

Re: Allocate length of word vs fixed length

Show all headers | View raw

On 19/07/2021 19:29, dfs wrote:
> On 7/19/21 2:01 PM, Bart wrote:
>> On 19/07/2021 17:50, dfs wrote:
>>> Loaded a list of words into an array.
>>>
>>> The 370103 words came from https://github.com/dwyl/english-words
>>> file = words_alpha.txt
>>>
>>> Tested a couple memory allocation 'strategies' during loading:
>>>
>>> 1. allocate the strlen() of each word
>>> 2. allocate a fixed length (len of longest word = 32)
>>>
>>> I figured getting strlen(word) each time would be slower than 
>>> allocating a fixed amt of memory, but that wasn't the case.
>>>
>>> Strategy 2 is significantly slower, and uses nearly 3x the memory.
> 
> Any thoughts on this?

Well, I must have loaded a different file from yours (words.zip), with 
466,000 words.

The size of that file on disk (and in memory using my method) is 4.8MB, 
so approx 10 bytes per word on average.

Allocating 32 bytes per word will use 3 times as much memory as you say 
(some 14MB)

The speed of it depends the access patterns, but clearly spreading it 
over an extra 9MB means 2/3 of the data loaded into cache mempry is useless.

Note that my method will use a total of 18 bytes per word on a 64-bit 
machine, with a 64-bit pointer per word. I haven't done any random 
access tests; I've concentrated on loading only.

Given that, there might be a way of using a fixed 16 bytes per word, not 
32, but you'd need some way of dealing with words longer than 15 
characters (nul is still needed). Whether that is going to be any 
faster, is hard to predict.

Maybe changing the order (from alphabetical) might help. It really 
depends on what you intend doing. What is needed are some benchmarks for 
an actual application that shows a problem.

There are all sorts of clever ways to arrange strings in memory, but 
some get very complicated.

> 
> 
>>> Also, does anyone have any 'tricks' to make such a file load routine 
>>> faster/more efficient?  Thanks
>>
>> How fast do you want it?
> 
> As fast as possible!

But is this loading the file (which doesn't appeart to be a problem on 
your machine), or doing something else with it?

> Amazing you wrote your own language.  Did you name it?

I usually call it 'M'.

Back to comp.lang.c | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Allocate length of word vs fixed length dfs <nospam@dfs.com> - 2021-07-19 12:50 -0400
  Re: Allocate length of word vs fixed length Bart <bc@freeuk.com> - 2021-07-19 19:01 +0100
    Re: Allocate length of word vs fixed length dfs <nospam@dfs.com> - 2021-07-19 14:29 -0400
      Re: Allocate length of word vs fixed length Bart <bc@freeuk.com> - 2021-07-19 19:49 +0100
        Re: Allocate length of word vs fixed length dfs <nospam@dfs.com> - 2021-07-19 18:01 -0400
          Re: Allocate length of word vs fixed length Bart <bc@freeuk.com> - 2021-07-19 23:25 +0100
      Re: Allocate length of word vs fixed length scott@slp53.sl.home (Scott Lurndal) - 2021-07-19 20:00 +0000
        Re: Allocate length of word vs fixed length Branimir Maksimovic <branimir.maksimovic@gmail.com> - 2021-07-19 23:29 +0000
      Re: Allocate length of word vs fixed length Real Troll <real.troll@trolls.com> - 2021-07-20 00:30 +0100
        Re: Allocate length of word vs fixed length DFS <nospam@dfs.com> - 2021-07-19 20:40 -0400
          Re: Allocate length of word vs fixed length Real Troll <real.troll@trolls.com> - 2021-07-20 17:00 +0000
      Re: Allocate length of word vs fixed length Kaz Kylheku <563-365-8930@kylheku.com> - 2021-07-24 01:07 +0000
  Re: Allocate length of word vs fixed length David Brown <david.brown@hesbynett.no> - 2021-07-19 21:35 +0200
    Re: Allocate length of word vs fixed length dfs <nospam@dfs.com> - 2021-07-19 18:02 -0400
  Re: Allocate length of word vs fixed length Ike Naar <ike@rie.sdf.org> - 2021-07-19 21:11 +0000
    Re: Allocate length of word vs fixed length dfs <nospam@dfs.com> - 2021-07-19 18:03 -0400
      Re: Allocate length of word vs fixed length Ike Naar <ike@sdf.org> - 2021-07-20 05:28 +0000
  Re: Allocate length of word vs fixed length dfs <nospam@dfs.com> - 2021-07-19 18:21 -0400
    Re: Allocate length of word vs fixed length Bart <bc@freeuk.com> - 2021-07-19 23:29 +0100
      Re: Allocate length of word vs fixed length dfs <nospam@dfs.com> - 2021-07-19 18:40 -0400
  Re: Allocate length of word vs fixed length Peter 'Shaggy' Haywood <phaywood@alphalink.com.au> - 2021-07-20 11:44 +1000
  Re: Allocate length of word vs fixed length Real Troll <real.troll@trolls.com> - 2021-07-20 22:15 +0000
    Re: Allocate length of word vs fixed length dfs <nospam@dfs.com> - 2021-07-20 20:22 -0400
      Re: Allocate length of word vs fixed length dfs <nospam@dfs.com> - 2021-07-21 23:23 -0400

csiph-web