Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.php > #9692 > unrolled thread

Exhaustive memory allocation using arrays

Back to article view | Back to comp.lang.php

Contents

  Exhaustive memory allocation using arrays tombert.at@gmail.com - 2013-05-28 23:27 -0700
    Re: Exhaustive memory allocation using arrays tombert.at@gmail.com - 2013-05-28 23:42 -0700
    Re: Exhaustive memory allocation using arrays tombert.at@gmail.com - 2013-05-29 00:45 -0700
      Re: Exhaustive memory allocation using arrays Christoph Becker <cmbecker69@gmx.de> - 2013-05-29 13:00 +0200
    Re: Exhaustive memory allocation using arrays Denis McMahon <denismfmcmahon@gmail.com> - 2013-05-29 08:22 +0000
      Re: Exhaustive memory allocation using arrays The Natural Philosopher <tnp@invalid.invalid> - 2013-05-29 09:54 +0100
    Re: Exhaustive memory allocation using arrays Thomas 'PointedEars' Lahn <PointedEars@web.de> - 2013-05-29 15:30 +0200

#9692 — Exhaustive memory allocation using arrays

Why is the following code using > 100MB?

for ($i=0; $i < 1000000; $i++) {
    $value[$i] = 23.4567890123;
}
echo (memory_get_usage(TRUE) / (1024 * 1024)) . "MB <br>";

It looks like that each item takes 100 Byte for allocation??

thx for your help!

[toc] | [next] | [standalone]

#9693

http://nikic.github.io/2011/12/12/How-big-are-PHP-arrays-really-Hint-BIG.html

[toc] | [prev] | [next] | [standalone]

#9694

And, doubling the memory size by assigning a new value:

for ($i=0; $i < 1000000; $i++) {
    $value[$i] = "23.4567890123";
}
for ($i=0; $i < 1000000; $i++) {
    $value[$i] = sprintf("%f", $value[$i]);
}

Now php is using ~400 MB

[toc] | [prev] | [next] | [standalone]

#9698

tombert.at wrote:
> And, doubling the memory size by assigning a new value:
> 
> for ($i=0; $i < 1000000; $i++) {
>     $value[$i] = "23.4567890123";
> }
> for ($i=0; $i < 1000000; $i++) {
>     $value[$i] = sprintf("%f", $value[$i]);
> }
> 
> Now php is using ~400 MB

I *assume* that the strings in the first loop do not actually allocate a
struct str for every instance, but instead reference a single statically
allocated instance.  The second loop is probably allocating a new struct
str for every sprintf().

Compare with:

  for ($i=0; $i < 1000000; $i++) {
      $value[$i] = "23.4567890123";
  }
  for ($i=0; $i < 1000000; $i++) {
      $value[$i] = floatval($value[$i]);
  }

-- 
Christoph M. Becker

[toc] | [prev] | [next] | [standalone]

#9695

On Tue, 28 May 2013 23:27:13 -0700, tombert.at wrote:

> Why is the following code using > 100MB?
> 
> for ($i=0; $i < 1000000; $i++) {
>     $value[$i] = 23.4567890123;
> }
> echo (memory_get_usage(TRUE) / (1024 * 1024)) . "MB <br>";
> 
> It looks like that each item takes 100 Byte for allocation??

From: http://php.net/manual/en/language.types.array.php

"An array in PHP is actually an ordered map."

You may be expecting arrays where an array of 1e6 floats is held in 
memory in 1e6 sequential 8 byte locations, for a total of 8e6 bytes, but 
I suspect that if you did, you'd be very very very wrong.

It's possible that:

$arr = array_values( $arr );

May shrink your array, I don't know for sure.

It's also possible that:

$a = Array();
for ( $i = 0; i < 1e6; i++ )
	$a[] = 23.4567890123;

might produce a smaller array, simply because it's not assigning keys. 
Again, I don't know for sure.

The reason I don't know for sure is that these implementation details 
are, to me, just that. If they don't affect me, I don't need to 
understand them in the level of detail you're looking for. There's plenty 
of things I do need to understand at that level of detail, and I have 
limited time to put into gaining such understandings, so I use it where 
it's most needed. :)

-- 
Denis McMahon, denismfmcmahon@gmail.com

[toc] | [prev] | [next] | [standalone]

#9696

On 29/05/13 09:22, Denis McMahon wrote:
> On Tue, 28 May 2013 23:27:13 -0700, tombert.at wrote:
>
>> Why is the following code using > 100MB?
>>
>> for ($i=0; $i < 1000000; $i++) {
>>      $value[$i] = 23.4567890123;
>> }
>> echo (memory_get_usage(TRUE) / (1024 * 1024)) . "MB <br>";
>>
>> It looks like that each item takes 100 Byte for allocation??
> From: http://php.net/manual/en/language.types.array.php
>
> "An array in PHP is actually an ordered map."
>
> You may be expecting arrays where an array of 1e6 floats is held in
> memory in 1e6 sequential 8 byte locations, for a total of 8e6 bytes, but
> I suspect that if you did, you'd be very very very wrong.
>
> It's possible that:
>
> $arr = array_values( $arr );
>
> May shrink your array, I don't know for sure.
>
> It's also possible that:
>
> $a = Array();
> for ( $i = 0; i < 1e6; i++ )
> 	$a[] = 23.4567890123;
>
> might produce a smaller array, simply because it's not assigning keys.
> Again, I don't know for sure.
>
> The reason I don't know for sure is that these implementation details
> are, to me, just that. If they don't affect me, I don't need to
> understand them in the level of detail you're looking for. There's plenty
> of things I do need to understand at that level of detail, and I have
> limited time to put into gaining such understandings, so I use it where
> it's most needed. :)
>
Exactly so. There are times when you realise that an 'excellent for some 
tasks' tool is in fact a complete dogs breakfast when applied to another 
task.

I found, for example, that on the fly generation of graphics using PHP 
was massively CPU and RAM hungry and appeared to suffer re-rentrancy 
problems when used in multiple concurrent access: the solution was to 
generate the image data asynchronously using a C program.

I can think of many ways in which an array element dynamically assigned 
needing pointers to memory blocks and so on could easily have several 
tens of bytes overhead.

That of course is an issue with an interpreted language: it has to more 
or less avoid statically allocated memory especially if there is no way 
to declare it.




-- 
Ineptocracy

(in-ep-toc’-ra-cy) – a system of government where the least capable to lead are elected by the least capable of producing, and where the members of society least likely to sustain themselves or succeed, are rewarded with goods and services paid for by the confiscated wealth of a diminishing number of producers.

[toc] | [prev] | [next] | [standalone]

#9702

tombert.at@gmail.com wrote:
^^^^^^^^^^^^^^^^^^^^
Please get a real name.

> Why is the following code using > 100MB?
> 
> for ($i=0; $i < 1000000; $i++) {
>     $value[$i] = 23.4567890123;
> }
> echo (memory_get_usage(TRUE) / (1024 * 1024)) . "MB <br>";
> 
> It looks like that each item takes 100 Byte for allocation??

First of all, 1 MB (megabyte) is 1'000'000 (one million, 10⁶) bytes.  What 
you call “MB” here actually is 1 _MiB_ (mega-binary byte; short: mebibyte) 
or 2²⁰ bytes = 1'048'576 bytes (as you divided by 2¹⁰ × 2¹⁰ = 2²⁰ to arrive 
at your “MB” value):

<http://en.wikipedia.org/wiki/Mebibyte>

So your calculation

  100 “MB” ∕ 1'000'000 items = 100 bytes∕item

is wrong to begin with.  Assuming that you got something close to 100 as a 
result of

  memory_get_usage(TRUE) / (1024 * 1024)

each item would take about

  104'857'600 bytes / 1'000'000 = 105 bytes

<http://php.net/memory_get_usage>

Second, the value you assigned is a value of the float type.  In PHP, the 
precision of the float type, thus the number of bits required to store such 
a value, is platform-dependent:

<http://php.net/manual/en/language.types.float.php>

The manual says that IEEE-754 “double precision” floating-point values are a 
common implementation (as they are in other equally dynamic programming 
languages), but we cannot assume that to be the case on your platform.  And 
you have not said which platform and PHP version you are using, so we cannot 
look into the pertinent source code.

Third, you are not just storing the float value.  You are storing a key-
value relationship:

,-<http://www.php.net/manual/en/language.types.array.php>
| 
| An array in PHP is actually an ordered map. A map is a type that
| associates values to keys.  […]

Fourth, that PHP arrays can be associative, i. e. can have string keys, and 
that using the string representation of a numeric key accesses the same item 
as the number (ibid.), suggests implementations in which all keys are 
actually string values (as it is in other equally dynamic programming 
languages, like ECMAScript implementations; if not like that, you would have 
to store the type of the key which is even more memory-expensive), and

,-<http://www.php.net/manual/en/language.types.string.php>
| 
| The string in PHP is implemented as an array of bytes and an integer
| indicating the length of the buffer.  […]

Fast access to array items further suggests that the keys are not only 
stored as-is (for array_keys()), but that the implementation includes a hash 
table.

Fifth, PHP arrays do have an internal pointer – <http://php.net/current> – 
that can be advanced: <http://php.net/next>.  Combined with the dynamic size 
of PHP arrays this suggests that the items of an array are at least arranged 
in a linked list.  The existence of a function to rewind the internal 
pointer – <http://php.net/prev> – further suggests that they are arranged in 
a double-linked list.

Finally, you are not just measuring the memory that is needed for the array, 
but for executing the preceding program.  There is *at least* the memory for 
$i variable to consider.  So “your math does not add up”.

Probably there are even more factors to consider.  See also the 
memory_get_usage() documentation and user comments.

So consider this: A PHP array could be implemented *at least* as a hash 
table (for quick item access) in which the string keys are stored in a 
linked list (so that array_keys() works), and the stored values could be in 
a double-linked list (so that internal pointer movements work).  In this 
hash table you are storing items whose plain values require *at least* 64 
bits = 8 bytes of memory each.  Add to that the memory required to run the 
program that fills the array.

I know of no native way to determine the memory it takes to store the PHP 
array; it can perhaps be obtained with a PHP profiler.  In any case, I 
presume it will be smaller than what you measured with memory_get_usage().


HTH

PointedEars
-- 
var bugRiddenCrashPronePieceOfJunk = (
    navigator.userAgent.indexOf('MSIE 5') != -1
    && navigator.userAgent.indexOf('Mac') != -1
)  // Plone, register_function.js:16

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.php

csiph-web