Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #8914 > unrolled thread

Not able to store data to dictionary because of memory limitation

Back to article view | Back to comp.lang.python

Contents

  Not able to store data to dictionary because of memory limitation Rama Rao Polneni <ramp99@gmail.com> - 2011-07-06 13:19 +0530
    Re: Not able to store data to dictionary because of memory limitation Ulrich Eckhardt <ulrich.eckhardt@dominolaser.com> - 2011-07-06 10:35 +0200
      Re: Not able to store data to dictionary because of memory limitation Rama Rao Polneni <ramp99@gmail.com> - 2011-07-31 14:08 +0530
      Re: Not able to store data to dictionary because of memory limitation Chris Angelico <rosuav@gmail.com> - 2011-07-31 09:46 +0100

#8914 — Not able to store data to dictionary because of memory limitation

Hi All,

I am facing a problem when I am storing cursor fetched(Oracle 10G)
data in to a dictionary.
As I don't have option to manipulate data in oracle10G, I had to stick
to python to parse the data for some metrics.

After storing 1.99 GB data in to the dictionary, python stopped to
store the remaining data in to dictionary.

Memory utilization is 26 GB/34GB. That means, still lot memory is left
as unutilized.

Can please share your experices/ideas to resolve this issue.
Is this prople mbecasue of large memory utlization.
Is there any alternate solution to resolve this issue. Like splitting
the dictionaries or writing the data to hard disk instead of writing
to memory.
How efficiently we can use  memory when we are going for dictionaries.

Thanks in advacne,
Rama

[toc] | [next] | [standalone]

#8922

Rama Rao Polneni wrote:
> After storing 1.99 GB data in to the dictionary, python stopped to
> store the remaining data in to dictionary.

Question here:
 - Which Python?
 - "stopped to store" (you mean "stopped storing", btw), how does it behave? 
Hang? Throw exceptions? Crash right away?


> Memory utilization is 26 GB/34GB. That means, still lot memory is left
> as unutilized.

2GiB is typically the process limit for memory allocations on 32-bit 
systems. So, if you are running a 32-bit system or running a 32-bit process 
on a 64-bit system, you are probably hitting hard limits. With luck, you 
could extend this to 3GiB on a 32-bit system.


> Is this proplem becasue of large memory utlization.

I guess yes.


> Is there any alternate solution to resolve this issue. Like splitting
> the dictionaries or writing the data to hard disk instead of writing
> to memory.

If you have lost of equal strings, interning them might help, both in size 
and speed. Doing in-memory compression would be a good choice, too, like 
e.g. if you have string fields in the DB that can only contain very few 
possible values, converting them to an integer/enumeration.

Otherwise, and this is a more general approach, prefer making a single sweep 
over the data. This means that you read a chunk of data, perform whatever 
operation you need on it, possibly write the results and then discard the 
chunk. This keeps memory requirements low. At first, it doesn't look as 
clean as reading the whole data in one step, calculations as a second and 
writing results as a third, but with such amounts of data as yours, it is 
the only viable step.

Good luck, and I'd like to hear how you solved the issue!

Uli

-- 
Domino Laser GmbH
Geschäftsführer: Thorsten Föcking, Amtsgericht Hamburg HR B62 932

[toc] | [prev] | [next] | [standalone]

#10621

Hi Ulrich,

Thanks for your idea.
I resolved the issue by making use of integers instead of strings.
Earlier I had many duplicate strings in different rows retrieved from database.
I created a list to have unique strings and their indexes are used in
actual computations. So lot of space is saved by having integers
instead of strings.

I  am trying differnt approach too, to get ids from database instead
of gettign direct values. values will be retrived by using unique ids
of different tables. I hope, I will be able to improve perforamnce
with this approach.

Regards,
Rama

On 7/6/11, Ulrich Eckhardt <ulrich.eckhardt@dominolaser.com> wrote:
> Rama Rao Polneni wrote:
>> After storing 1.99 GB data in to the dictionary, python stopped to
>> store the remaining data in to dictionary.
>
> Question here:
>  - Which Python?
>  - "stopped to store" (you mean "stopped storing", btw), how does it behave?
> Hang? Throw exceptions? Crash right away?
>
>
>> Memory utilization is 26 GB/34GB. That means, still lot memory is left
>> as unutilized.
>
> 2GiB is typically the process limit for memory allocations on 32-bit
> systems. So, if you are running a 32-bit system or running a 32-bit process
> on a 64-bit system, you are probably hitting hard limits. With luck, you
> could extend this to 3GiB on a 32-bit system.
>
>
>> Is this proplem becasue of large memory utlization.
>
> I guess yes.
>
>
>> Is there any alternate solution to resolve this issue. Like splitting
>> the dictionaries or writing the data to hard disk instead of writing
>> to memory.
>
> If you have lost of equal strings, interning them might help, both in size
> and speed. Doing in-memory compression would be a good choice, too, like
> e.g. if you have string fields in the DB that can only contain very few
> possible values, converting them to an integer/enumeration.
>
> Otherwise, and this is a more general approach, prefer making a single sweep
> over the data. This means that you read a chunk of data, perform whatever
> operation you need on it, possibly write the results and then discard the
> chunk. This keeps memory requirements low. At first, it doesn't look as
> clean as reading the whole data in one step, calculations as a second and
> writing results as a third, but with such amounts of data as yours, it is
> the only viable step.
>
> Good luck, and I'd like to hear how you solved the issue!
>
> Uli
>
> --
> Domino Laser GmbH
> Geschäftsführer: Thorsten Föcking, Amtsgericht Hamburg HR B62 932
>
> --
> http://mail.python.org/mailman/listinfo/python-list
>

[toc] | [prev] | [next] | [standalone]

#10622

On Sun, Jul 31, 2011 at 9:38 AM, Rama Rao Polneni <ramp99@gmail.com> wrote:
> Earlier I had many duplicate strings in different rows retrieved from database.
> I created a list to have unique strings and their indexes are used in
> actual computations. So lot of space is saved by having integers
> instead of strings.
>

What you're doing there is called "interning" the strings, and I think
it's the right thing to do. You can get the Python interpreter to do
this for you by simply passing the strings through the built-in
function intern() which will return a string with identical content.
(In Python 3, that function has been shoved off to a module, so it's
sys.intern() and you need to import sys.) You'll save some space and
improve dictionary performance, but you won't lose clarity.

Chris Angelico

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web