Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]


Groups > comp.soft-sys.math.mathematica > #4057 > unrolled thread

Memory leak or flawed garbage collector

Started by"Fabrice P. Laussy" <fabrice.laussy@gmail.com>
First post2011-07-29 12:05 +0000
Last post2011-08-12 09:11 +0000
Articles 9 — 4 participants

Back to article view | Back to comp.soft-sys.math.mathematica


Contents

  Memory leak or flawed garbage collector "Fabrice P. Laussy" <fabrice.laussy@gmail.com> - 2011-07-29 12:05 +0000
    Re: Memory leak or flawed garbage collector "Oleksandr Rasputinov" <oleksandr_rasputinov@hmamail.com> - 2011-07-30 10:04 +0000
      Re: Memory leak or flawed garbage collector "Alexey Popkov" <lehin.p@gmail.com> - 2011-07-31 11:27 +0000
      Re: Memory leak or flawed garbage collector "Oleksandr Rasputinov" <oleksandr_rasputinov@hmamail.com> - 2011-08-01 03:38 +0000
        Re: Memory leak or flawed garbage collector "Alexey Popkov" <lehin.p@gmail.com> - 2011-08-02 11:12 +0000
        Re: Memory leak or flawed garbage collector Daniel Lichtblau <danl@wolfram.com> - 2011-08-03 11:16 +0000
      Re: Memory leak or flawed garbage collector "Oleksandr Rasputinov" <oleksandr_rasputinov@hmamail.com> - 2011-08-03 11:15 +0000
      Re: Memory leak or flawed garbage collector "Alexey Popkov" <lehin.p@gmail.com> - 2011-08-11 09:19 +0000
      Re: Memory leak or flawed garbage collector "Oleksandr Rasputinov" <oleksandr_rasputinov@hmamail.com> - 2011-08-12 09:11 +0000

#4057 — Memory leak or flawed garbage collector

From"Fabrice P. Laussy" <fabrice.laussy@gmail.com>
Date2011-07-29 12:05 +0000
SubjectMemory leak or flawed garbage collector
Message-ID<j0u7mo$fc1$1@smc.vnet.net>
Dear Group,

I have a problem of garbage collection or of memory leak, with a
module that will seemingly not discard its internal variables, thus
leading to explosion of memory usage by repeated calls to this module.

Opening the notebook, nothing else being done yet:

	In[1]:= MemoryInUse[]
	Out[1]= 18460176

Then I define the said module, with private variables, etc. Before
running the first instance of it, already there appears many
variables, some defined globally, others with the $ of internal
variables:

	?Global`*
	
	Global`
	A	coef	Disa$	g2	K\[Sigma]3$	L\[Sigma]1	mybasis	n\[Sigma]3	re$	X	\[Gamma]	\[Nu]\[Nu]	\[Omega]3$
	AllM	coefa	Dis\[Sigma]1	g3	L	L\[Sigma]1$	mybasis$	n\[Sigma]3$	solMean	Xsol	\[Gamma]1	\[Nu]\[Nu]\[Nu]	\[Omega]a
	AllM$	coefa$	Dis\[Sigma]1$	i	La	L\[Sigma]2	myvar	n$	solMean$	Xsol$	\[Gamma]2	\[Nu]\[Nu]\[Nu]$	\[Omega]a$
	Av	coef\[Sigma]1	eig	im	La$	L\[Sigma]2$	m\[CapitalDelta]	order	Spea	x$	\[Gamma]3	\[Nu]\[Nu]$	\[Omega]max
	Average	coef\[Sigma]1$	eigV	im$	lcor	L\[Sigma]3	m$	out	Spea$	X$	\[Gamma]a	\[Nu]$	\[Omega]min
	Average$	coef\[Sigma]2	eigV$	Ind	lcor$	L\[Sigma]3$	M$	out$	Spectra3dots	y	\[CapitalDelta]	\[CapitalSigma]1	
	Av$	coef\[Sigma]2$	eig$	Ind$	lmean	L$	n	P	Spe\[Sigma]1	y$	\[CapitalDelta]1	\[CapitalSigma]1$	
	A$	coef\[Sigma]3	eme	i$	lmean0	m	na	P1	Spe\[Sigma]1$	\[Alpha]	\[CapitalDelta]2	\[CapitalSigma]2	
	broad	coef\[Sigma]3$	emeB	j	lmean0$	M	na$	P2	Spe\[Sigma]2	\[Alpha]a	\[CapitalDelta]3	\[CapitalSigma]2$	
	broad$	coef$	emeB$	j$	lmean$	M1	nmax	P3	Spe\[Sigma]2$	\[Alpha]a$	\[Delta]\[Omega]	\[CapitalSigma]3	
	c	Corr	eme$	Ka	Lora	M1$	nn	Pa	Spe\[Sigma]3	\[Alpha]\[Sigma]1	\[Mu]	\[CapitalSigma]3$	
	case	CorrSpec	ene	Ka$	Lora$	M2	nn$	re	Spe\[Sigma]3$	\[Alpha]\[Sigma]1$	\[Mu]\[Mu]	\[Omega]	
	Change	CorrSpec$	ene$	K\[Sigma]1	Lor\[Sigma]1	M2$	nt	regmat	Tota	\[Alpha]\[Sigma]2	\[Mu]\[Mu]\[Mu]	\[Omega]1	
	ChangeBack	Corr$	f1	K\[Sigma]1$	Lor\[Sigma]1$	Me	n\[Sigma]1	RegMat3dots	Tota$	\[Alpha]\[Sigma]2$	\[Mu]\[Mu]\[Mu]$	
\[Omega]1$	
	ChangeBack$	Co$	f1$	K\[Sigma]2	ls	Me$	n\[Sigma]1$	RegMatrix	Tot\[Sigma]1	\[Alpha]\[Sigma]3	\[Mu]\[Mu]$	\[Omega]2	
	Change$	c$	fs	K\[Sigma]2$	ls$	mm	n\[Sigma]2	RegMatrix$	Tot\[Sigma]1$	\[Alpha]\[Sigma]3$	\[Mu]$	\[Omega]2$	
	Co	Disa	g1	K\[Sigma]3	l\[CapitalDelta]1	mm$	n\[Sigma]2$	regmat$	x	\[Alpha]$	\[Nu]	\[Omega]3	

After running the first instance of the module:

	In[32]:= MemoryInUse[]
	Out[32]= 26536288

After running the second instance:

        In[35]:= MemoryInUse[]
        Out[35]= 32878688

Etc., the memory will keep increasing, although each module runs a
separate case and could be computed in separate notebooks, thus the
amount of required memory should be the same.

The variables indeed keep cluttering things from the various instances
of the module:

	?Global`*
	
	 Global`
	A	coef\[Sigma]1	eig$	i$	lcor$	m	n	P3	Spe\[Sigma]1	\[Alpha]a$	\[Mu]\[Mu]\[Mu]	\[Omega]1$632
	AllM	coef\[Sigma]1$	eme	i$646	lcor$646	M	na	Pa	Spe\[Sigma]1$	\[Alpha]\[Sigma]1	\[Mu]\[Mu]\[Mu]$	\[Omega]1$647
	AllM$	coef\[Sigma]2	emeB	j	lmean	M1	na$	re	Spe\[Sigma]2	\[Alpha]\[Sigma]1$	\[Mu]\[Mu]$	\[Omega]1$648
	Av	coef\[Sigma]2$	emeB$	j$	lmean0	M1$	nmax	regmat	Spe\[Sigma]2$	\[Alpha]\[Sigma]2	\[Mu]$	\[Omega]2
	Average	coef\[Sigma]3	eme$	Ka	lmean0$	M2	nn	RegMat3dots	Spe\[Sigma]3	\[Alpha]\[Sigma]2$	\[Nu]	\[Omega]2$
	Average$	coef\[Sigma]3$	ene	Ka$	lmean$	M2$	nn$	RegMatrix	Spe\[Sigma]3$	\[Alpha]\[Sigma]3	\[Nu]\[Nu]	\[Omega]2$630
	Av$	coef$	ene$	Ka$629	Lora	Me	nt	RegMatrix$	Tota	\[Alpha]\[Sigma]3$	\[Nu]\[Nu]\[Nu]	\[Omega]2$632
	A$	Corr	f1	Ka$646	Lora$	Me$	n\[Sigma]1	RegMatrix$630	Tota$	\[Alpha]$	\[Nu]\[Nu]\[Nu]$	\[Omega]2$647
	broad	CorrSpec	f1$	K\[Sigma]1	Lor\[Sigma]1	mm	n\[Sigma]1$	RegMatrix$632	Tot\[Sigma]1	\[Gamma]	\[Nu]\[Nu]$	
\[Omega]2$648
	broad$	CorrSpec$	fs	K\[Sigma]1$	Lor\[Sigma]1$	mm$	n\[Sigma]2	RegMatrix$647	Tot\[Sigma]1$	\[Gamma]1	\[Nu]$	
\[Omega]3
	c	Corr$	g1	K\[Sigma]2	ls	mybasis	n\[Sigma]2$	RegMatrix$648	x	\[Gamma]2	\[CapitalSigma]1	\[Omega]3$
	case	Co$	g2	K\[Sigma]2$	ls$	mybasis$	n\[Sigma]3	regmat$	X	\[Gamma]3	\[CapitalSigma]1$	\[Omega]a
	Change	c$	g3	K\[Sigma]3	l\[CapitalDelta]1	myvar	n\[Sigma]3$	re$	Xsol	\[Gamma]a	\[CapitalSigma]2	\[Omega]a$
	ChangeBack	Disa	i	K\[Sigma]3$	L\[Sigma]1	m\[CapitalDelta]	n$	re$629	Xsol$	\[CapitalDelta]	\[CapitalSigma]2$	\[Omega]a$630
	ChangeBack$	Disa$	im	L	L\[Sigma]1$	m$	order	re$646	x$	\[CapitalDelta]1	\[CapitalSigma]3	\[Omega]a$632
	Change$	Dis\[Sigma]1	im$	La	L\[Sigma]2	M$	out	solMean	X$	\[CapitalDelta]2	\[CapitalSigma]3$	\[Omega]a$647
	Co	Dis\[Sigma]1$	im$629	La$	L\[Sigma]2$	M$630	out$	solMean$	y	\[CapitalDelta]3	\[Omega]	\[Omega]a$648
	coef	eig	im$646	La$629	L\[Sigma]3	M$632	P	Spea	y$	\[Delta]\[Omega]	\[Omega]1	\[Omega]max
	coefa	eigV	Ind	La$646	L\[Sigma]3$	M$647	P1	Spea$	\[Alpha]	\[Mu]	\[Omega]1$	\[Omega]min
	coefa$	eigV$	Ind$	lcor	L$	M$648	P2	Spectra3dots	\[Alpha]a	\[Mu]\[Mu]	\[Omega]1$630	

Now, if we take for instance, RegMatrix$632, which appears in the list above. It seems to take no memory:

     	In[39]:= ByteCount[RegMatrix$632]
	Out[39]= 0

But it clearly is stored in memory:

	?RegMatrix$632
	Global`RegMatrix$632
	Attributes[RegMatrix$632]={Temporary}     
	
	RegMatrix$632[{0,0,0,0,0,0,0,0},{-1,-1,0,0,0,0,0,0}]=0.
	RegMatrix$632[{0,0,0,0,0,0,0,0},{-1,0,0,0,0,0,1,0}]=0
	RegMatrix$632[{0,0,0,0,0,0,0,0},{-1,0,0,0,1,0,0,0}]=0
	
	[snip]
	
(a lot of more stuff here), and if I remove it:

	In[47]:= MemoryInUse[]
	Out[47]= 32879624
	
	In[48]:= Remove[RegMatrix$632]

	In[49]:= MemoryInUse[]
	Out[49]= 29521760

So clearly, along with all the other stuff lying around, it accounts
for this filling up of the memory. It seems there is a problem with
garbage collection and disposing of local variables defined within a
module.

I don't think there is anything special with the module itself. It
calls other modules defined externally. At the same time the problem
does not seem reproducible defining a toy-module and looking for the
same behaviour. Does this problem seem familiar? Strange? What could
be causing it?

[toc] | [next] | [standalone]


#4062

From"Oleksandr Rasputinov" <oleksandr_rasputinov@hmamail.com>
Date2011-07-30 10:04 +0000
Message-ID<j10l01$os6$1@smc.vnet.net>
In reply to#4057
Dear Fabrice

Mathematica's garbage collection is based on reference counting. Thus, if  
one creates a reference to a Temporary symbol, whether directly or  
indirectly, the symbol cannot be garbage-collected until the reference is  
removed. Here is an example:

In[1] :=
Module[{a}, a]

Out[1] =
a$592

In[2] :=
Names["Global`*"]

Out[2] =
{a, a$592}

Thus we see that the symbol Out has a DownValue (i.e. Out[1]) which  
references the internal symbol a$592. This reference is counted and thus  
prevents garbage collection. To clear it up we may do the following:

In[3] :=
Unprotect[Out];
Clear[Out];
Protect[Out];

In[6] :=
Names["Global`*"]

Out[6] =
{a}

Of course, it is quite possible that the references to your symbols  
responsible for their not being garbage-collected are somewhere other than  
the DownValues of In or Out, but if you find and remove these, you should  
find that the symbols are garbage-collected (unless for some reason they  
have had their Temporary attribute removed during the course of  
execution). If this does not occur after all references have been removed,  
then indeed a bug in the garbage collector would indeed be a possibility.  
(It is conceivable that references are not counted immediately, as would  
be the case in a mark-sweep garbage collector, but as far as I know such  
mechanisms are not in fact employed as far as the interpreter is  
concerned; they would likely be rather difficult to implement in a  
term-rewriting language such as Mathematica.)

Best,

O. R.

On Fri, 29 Jul 2011 13:05:44 +0100, Fabrice P. Laussy  
<fabrice.laussy@gmail.com> wrote:

> Dear Group,
>
> I have a problem of garbage collection or of memory leak, with a
> module that will seemingly not discard its internal variables, thus
> leading to explosion of memory usage by repeated calls to this module.
>
> Opening the notebook, nothing else being done yet:
>
> 	In[1]:= MemoryInUse[]
> 	Out[1]= 18460176
>
> Then I define the said module, with private variables, etc. Before
> running the first instance of it, already there appears many
> variables, some defined globally, others with the $ of internal
> variables:
>
> 	?Global`*
> 	
> 	Global`
> 	<many symbols>
>
> After running the first instance of the module:
>
> 	In[32]:= MemoryInUse[]
> 	Out[32]= 26536288
>
> After running the second instance:
>
>         In[35]:= MemoryInUse[]
>         Out[35]= 32878688
>
> Etc., the memory will keep increasing, although each module runs a
> separate case and could be computed in separate notebooks, thus the
> amount of required memory should be the same.
>
> The variables indeed keep cluttering things from the various instances
> of the module:
>
> 	?Global`*
> 	
> 	Global`
>       <more symbols>
>
> Now, if we take for instance, RegMatrix$632, which appears in the list  
> above. It seems to take no memory:
>
>      	In[39]:= ByteCount[RegMatrix$632]
> 	Out[39]= 0
>
> But it clearly is stored in memory:
>
> 	?RegMatrix$632
> 	Global`RegMatrix$632
> 	Attributes[RegMatrix$632]={Temporary}
> 	
> 	RegMatrix$632[{0,0,0,0,0,0,0,0},{-1,-1,0,0,0,0,0,0}]=0.
> 	RegMatrix$632[{0,0,0,0,0,0,0,0},{-1,0,0,0,0,0,1,0}]=0
> 	RegMatrix$632[{0,0,0,0,0,0,0,0},{-1,0,0,0,1,0,0,0}]=0
> 	
> 	[snip]
> 	
> (a lot of more stuff here), and if I remove it:
>
> 	In[47]:= MemoryInUse[]
> 	Out[47]= 32879624
> 	
> 	In[48]:= Remove[RegMatrix$632]
>
> 	In[49]:= MemoryInUse[]
> 	Out[49]= 29521760
>
> So clearly, along with all the other stuff lying around, it accounts
> for this filling up of the memory. It seems there is a problem with
> garbage collection and disposing of local variables defined within a
> module.
>
> I don't think there is anything special with the module itself. It
> calls other modules defined externally. At the same time the problem
> does not seem reproducible defining a toy-module and looking for the
> same behaviour. Does this problem seem familiar? Strange? What could
> be causing it?

[toc] | [prev] | [next] | [standalone]


#4091

From"Alexey Popkov" <lehin.p@gmail.com>
Date2011-07-31 11:27 +0000
Message-ID<j13e6q$54f$1@smc.vnet.net>
In reply to#4062
"Oleksandr Rasputinov" <oleksandr_rasputinov@hmamail.com> wrote:
news:j10l01$os6$1@smc.vnet.net...
> Of course, it is quite possible that the references to your symbols
> responsible for their not being garbage-collected are somewhere other than
> the DownValues of In or Out, but if you find and remove these, you should
> find that the symbols are garbage-collected (unless for some reason they
> have had their Temporary attribute removed during the course of
> execution). If this does not occur after all references have been removed,
> then indeed a bug in the garbage collector would indeed be a possibility.

There is a bug in the garbage collector (non-removing unreferenced temporary
variables with their definitions):

In[1]:= $HistoryLength=0;
a[b_]:=Module[{c,d},d:=9;d/;b===1];
Length@Names[$Context<>"*"]

Out[3]= 6

In[4]:= lst=Table[a[1],{1000}];
Length@Names[$Context<>"*"]

Out[5]= 1007

In[6]:= lst=.
Length@Names[$Context<>"*"]

Out[7]= 1007

In[8]:= Definition@d$999

Out[8]= Attributes[d$999]={Temporary}

d$999:=9


This bug was originally reported in this MathGroup post:
http://forums.wolfram.com/mathgroup/archive/2010/Dec/msg00130.html

[toc] | [prev] | [next] | [standalone]


#4116

From"Oleksandr Rasputinov" <oleksandr_rasputinov@hmamail.com>
Date2011-08-01 03:38 +0000
Message-ID<j15738$b4a$1@smc.vnet.net>
In reply to#4062
On Sun, 31 Jul 2011 12:27:22 +0100, Alexey Popkov <lehin.p@gmail.com>  
wrote:

> "Oleksandr Rasputinov" <oleksandr_rasputinov@hmamail.com> wrote:
> news:j10l01$os6$1@smc.vnet.net...
>> Of course, it is quite possible that the references to your symbols
>> responsible for their not being garbage-collected are somewhere other  
>> than
>> the DownValues of In or Out, but if you find and remove these, you  
>> should
>> find that the symbols are garbage-collected (unless for some reason they
>> have had their Temporary attribute removed during the course of
>> execution). If this does not occur after all references have been  
>> removed,
>> then indeed a bug in the garbage collector would indeed be a  
>> possibility.
>
> There is a bug in the garbage collector (non-removing unreferenced  
> temporary
> variables with their definitions):
>
> In[1]:= $HistoryLength=0;
> a[b_]:=Module[{c,d},d:=9;d/;b===1];
> Length@Names[$Context<>"*"]
>
> Out[3]= 6
>
> In[4]:= lst=Table[a[1],{1000}];
> Length@Names[$Context<>"*"]
>
> Out[5]= 1007
>
> In[6]:= lst=.
> Length@Names[$Context<>"*"]
>
> Out[7]= 1007
>
> In[8]:= Definition@d$999
>
> Out[8]= Attributes[d$999]={Temporary}
>
> d$999:=9
>
>
> This bug was originally reported in this MathGroup post:
> http://forums.wolfram.com/mathgroup/archive/2010/Dec/msg00130.html

Interesting. The bug does not manifest itself if Set is used instead of  
SetDelayed:

In[1] :=
$HistoryLength=0;
a[b_]:=Module[{c,d},d=9;d/;b===1];
Length@Names[$Context<>"*"]

Out[3] =
6

In[4] :=
lst=Table[a[1],{1000}];
Length@Names[$Context<>"*"]

Out[5] =
7 (* the new one is lst *)

One may speculate that this is due to special treatment of delayed values.  
In a new session:

In[1] :=
$HistoryLength=0;
a[b_]:=Module[{c,d},d:=9;d/;b===1];
Length@Names[$Context<>"*"]

Out[3] =
6

In[4] :=
a[1];
Names[$Context<>"*"]

Out[5] =
{"a", "b", "c", "c$", "d", "d$", "d$539"}

In[6] :=
OwnValues[d$539]

Out[6] =
{HoldPattern[d$539] :> 9}

In[7] :=
test = 1;
OwnValues[test] (* for comparison *)

Out[8] =
{HoldPattern[test] :> 1} (* immediate values do not look any different to  
delayed values from this point of view. Delayed values may therefore be  
distinguished by an internal flag; the fact that the bug is not present  
for immediate values implies that it is the delayed values rather than the  
immediate ones that are subject to special treatment. *)

So, let us try:

In[9] :=
d$539 =.;
Names[$Context <> "*"]

Out[10] =
{"a", "b", "c", "c$", "d", "d$", "test"} (* the symbol has now been  
garbage-collected *)

 From this we may surmise that values are not being cleared for symbols  
appearing in a Module that have gone out of scope if the delayed flag is  
set and if they have also been conditionally evaluated. It appears that  
these values are what is preventing garbage collection; therefore I would  
consider this as a bug in Module (in that it does not recognise when  
values have gone out of scope in all cases) rather than in the garbage  
collector itself.

As reported in the original thread, the same behaviour is not evident with  
Block, which is logically equivalent to Module in this simple case. Block  
also does not need to create and maintain temporary symbols, and therefore  
offers better performance than Module; it seems that this kind of lexical  
scoping is merely emulated in Mathematica, with dynamic scoping fitting  
more naturally with the language semantics. For these reasons, unless I  
explicitly require the temporary symbols, I generally prefer Block in most  
simple cases, or With where lexical scoping is called for.

[toc] | [prev] | [next] | [standalone]


#4129

From"Alexey Popkov" <lehin.p@gmail.com>
Date2011-08-02 11:12 +0000
Message-ID<j18m35$sls$1@smc.vnet.net>
In reply to#4116
"Oleksandr Rasputinov" <oleksandr_rasputinov@hmamail.com> wrote:
news:j15738$b4a$1@smc.vnet.net...
>  From this we may surmise that values are not being cleared for symbols
> appearing in a Module that have gone out of scope if the delayed flag is
> set and if they have also been conditionally evaluated. It appears that
> these values are what is preventing garbage collection; therefore I would
> consider this as a bug in Module (in that it does not recognise when
> values have gone out of scope in all cases) rather than in the garbage
> collector itself.

Consider the following simple experiment without Module:

In[1]:= $HistoryLength=0;
a[b_]:=(SetAttributes[d,Temporary];d:=9;d/;b===1);

In[3]:= a[1]
Out[3]= 9

In[4]:= Definition[d]
Out[4]= Attributes[d]={Temporary}

d:=9

As you see, the same bug appears without Module. So I think this is a bug in
the garbage collector itself. As for Module, I feel that this function is
essentially top-level, as opposed to Block. Probably it is possible to write
full analog of Module in top-level Mathematica code.

> Block
> also does not need to create and maintain temporary symbols, and therefore
> offers better performance than Module; it seems that this kind of lexical
> scoping is merely emulated in Mathematica, with dynamic scoping fitting
> more naturally with the language semantics. For these reasons, unless I
> explicitly require the temporary symbols, I generally prefer Block in most
> simple cases, or With where lexical scoping is called for.

Interesting. I agree that Module looks a bit hacky from the point of view of
the overall design of the system. As Richard Fateman wrote in his 1991 year
paper "A review of Mathematica", Module was introduced in version 2 for
fixing inconsistent design of Block scoping and scoping in replacement
rules: defining
g[x_] := Block[{a}, a = 1 + x]
and then evaluating g[a] gives $RecursionLimit::"reclim" error instead of
1+a. Module solves this problem.

[toc] | [prev] | [next] | [standalone]


#4160

FromDaniel Lichtblau <danl@wolfram.com>
Date2011-08-03 11:16 +0000
Message-ID<j1balr$ao9$1@smc.vnet.net>
In reply to#4116
On 08/02/2011 06:11 AM, Alexey Popkov wrote:
> "Oleksandr Rasputinov"<oleksandr_rasputinov@hmamail.com>  wrote:
> news:j15738$b4a$1@smc.vnet.net...
>>    From this we may surmise that values are not being cleared for symbols
>> appearing in a Module that have gone out of scope if the delayed flag is
>> set and if they have also been conditionally evaluated. It appears that
>> these values are what is preventing garbage collection; therefore I would
>> consider this as a bug in Module (in that it does not recognise when
>> values have gone out of scope in all cases) rather than in the garbage
>> collector itself.
>
> Consider the following simple experiment without Module:
>
> In[1]:= $HistoryLength=0;
> a[b_]:=(SetAttributes[d,Temporary];d:=9;d/;b===1);
>
> In[3]:= a[1]
> Out[3]= 9
>
> In[4]:= Definition[d]
> Out[4]= Attributes[d]={Temporary}
>
> d:=9
>
> As you see, the same bug appears without Module. So I think this is a bug in
> the garbage collector itself. As for Module, I feel that this function is
> essentially top-level, as opposed to Block. Probably it is possible to write
> full analog of Module in top-level Mathematica code.
> [...]

It does not require a condition test to elicit this behavior.

In[3]:= $HistoryLength = 0;
a[] := (SetAttributes[d, Temporary]; d := 9; d);

In[5]:= a[]

Out[5]= 9

In[6]:= ?? d

Global`d

Attributes[d]={Temporary}

d:=9

This is probably not incorrect behavior. Setting a Temporary attribute 
is not documented to have any effect, so far as I can tell. Module will 
set the attribute on its parameters, but that's a different matter.


Daniel Lichtblau
Wolfram Research

[toc] | [prev] | [next] | [standalone]


#4159

From"Oleksandr Rasputinov" <oleksandr_rasputinov@hmamail.com>
Date2011-08-03 11:15 +0000
Message-ID<j1bakt$ao4$1@smc.vnet.net>
In reply to#4062
On Tue, 02 Aug 2011 12:12:37 +0100, Alexey Popkov <lehin.p@gmail.com>  
wrote:

> "Oleksandr Rasputinov" <oleksandr_rasputinov@hmamail.com> wrote:
> news:j15738$b4a$1@smc.vnet.net...
>>  From this we may surmise that values are not being cleared for symbols
>> appearing in a Module that have gone out of scope if the delayed flag is
>> set and if they have also been conditionally evaluated. It appears that
>> these values are what is preventing garbage collection; therefore I  
>> would
>> consider this as a bug in Module (in that it does not recognise when
>> values have gone out of scope in all cases) rather than in the garbage
>> collector itself.
>
> Consider the following simple experiment without Module:
>
> In[1]:= $HistoryLength=0;
> a[b_]:=(SetAttributes[d,Temporary];d:=9;d/;b===1);
>
> In[3]:= a[1]
> Out[3]= 9
>
> In[4]:= Definition[d]
> Out[4]= Attributes[d]={Temporary}
>
> d:=9
>
> As you see, the same bug appears without Module. So I think this is a  
> bug in the garbage collector itself.

There is certainly some subtle behaviour going on here. The reason d is  
still around at the end of this is that it appears lexically in the input  
and is thus referred to by a. For example:

In[1] :=
$HistoryLength = 0;
x[y_] := (
    (* to avoid referencing q explicitly *)
    SetAttributes[Evaluate@Symbol["q"], Temporary];
   );

In[3]:=
Names[$Context <> "*"]

Out[3]=
{"x", "y"}

In[4] :=
x[1];
Names[$Context <> "*"]

Out[5] =
{"x", "y"}

So as long as q isn't referred to directly, the garbage collector does its  
job and the symbol is removed immediately. (This is somewhat useless for  
practical purposes however as the symbol is created and destroyed so  
quickly that one cannot do anything with it.) On the other hand, in a new  
session:

In[1] :=
$HistoryLength = 0;
a[b_] := SetAttributes[d, Temporary];

In[3] :=
Names[$Context <> "*"]

Out[3] =
{"a", "b", "d"} (* even though we haven't done anything yet; reference  
count = 1 *)

In[4] :=
a[1];
Names[$Context <> "*"]

Out[4] =
{"a", "b", "d"} (* no change; reference count still 1 *)

In[5] :=
Clear[a];
Names[$Context <> "*"]

Out[6] =
{"a", "b"} (* reference count 0; it's gone *)

So far all is as expected. But, in a new session:

$HistoryLength = 0;
a[b_] := (
    SetAttributes[d, Temporary];
    d := 9
    (* we don't bother with the conditional evaluation *)
   );

In[3] :=
a[1];
Names[$Context <> "*"]

Out[4] =
{"a", "b", "d"} (* still as expected, per above *)

In[5] :=
Clear[a];
Names[$Context <> "*"]

Out[6] =
{"a", "b", "d"} (* ??? *)

In[7] :=
d = 9; (* delayed assignment changed to immediate *)
Names[$Context <> "*"]

Out[8] =
{"a", "b"} (* gone *)

So it seems that the conjectured "delayed flag" set by SetDelayed  
constitutes a reference to the symbol which prevents its being  
garbage-collected. Needless to say this is rather obscure from the user's  
point of view and could be considered a bug on that account, though there  
is presumably a justifiable reason for it. And, of course, you are correct  
that it is nothing to do with Module per se (which merely confuses the  
issue).

> As for Module, I feel that this function is
> essentially top-level, as opposed to Block. Probably it is possible to  
> write full analog of Module in top-level Mathematica code.

Yes, I agree.

>> Block
>> also does not need to create and maintain temporary symbols, and  
>> therefore
>> offers better performance than Module; it seems that this kind of  
>> lexical
>> scoping is merely emulated in Mathematica, with dynamic scoping fitting
>> more naturally with the language semantics. For these reasons, unless I
>> explicitly require the temporary symbols, I generally prefer Block in  
>> most
>> simple cases, or With where lexical scoping is called for.
>
> Interesting. I agree that Module looks a bit hacky from the point of  
> view of
> the overall design of the system. As Richard Fateman wrote in his 1991  
> year
> paper "A review of Mathematica", Module was introduced in version 2 for
> fixing inconsistent design of Block scoping and scoping in replacement
> rules: defining
> g[x_] := Block[{a}, a = 1 + x]
> and then evaluating g[a] gives $RecursionLimit::"reclim" error instead of
> 1+a. Module solves this problem.

I'm not sure if I would call this "inconsistent". It is inconvenient, and  
perhaps unexpected to the uninitiated, but it seems to me essentially a  
result of mixing together constructs that imply lexical and dynamic  
scoping in a conflicting way. Module is, however, a good solution to the  
difficulties thus encountered.

[toc] | [prev] | [next] | [standalone]


#4319

From"Alexey Popkov" <lehin.p@gmail.com>
Date2011-08-11 09:19 +0000
Message-ID<j206ri$76q$1@smc.vnet.net>
In reply to#4062
There is another old bug with non-deleting unreferenced Module variables 
after Part assignments to them in v.5.2 which is not completely fixed even 
in version 7.0.1:

In[1]:= $HistoryLength=0;$Version
Module[{L=Array[0&,10^7]},L[[#]]++&/@Range[100];];
Names["L$*"]
ByteCount@Symbol@#&/@Names["L$*"]
Out[1]= 7.0 for Microsoft Windows (32-bit) (February 18, 2009)
Out[3]= {L$111}
Out[4]= {40000084}

Ref.: http://forums.wolfram.com/mathgroup/archive/2006/Mar/msg00170.html 

[toc] | [prev] | [next] | [standalone]


#4348

From"Oleksandr Rasputinov" <oleksandr_rasputinov@hmamail.com>
Date2011-08-12 09:11 +0000
Message-ID<j22qol$gsn$1@smc.vnet.net>
In reply to#4062
On Thu, 11 Aug 2011 10:19:46 +0100, Alexey Popkov <lehin.p@gmail.com>  
wrote:

> There is another old bug with non-deleting unreferenced Module variables
> after Part assignments to them in v.5.2 which is not completely fixed  
> even
> in version 7.0.1:
>
> In[1]:= $HistoryLength=0;$Version
> Module[{L=Array[0&,10^7]},L[[#]]++&/@Range[100];];
> Names["L$*"]
> ByteCount@Symbol@#&/@Names["L$*"]
> Out[1]= 7.0 for Microsoft Windows (32-bit) (February 18, 2009)
> Out[3]= {L$111}
> Out[4]= {40000084}
>
> Ref.: http://forums.wolfram.com/mathgroup/archive/2006/Mar/msg00170.html
>

Nor in 8.0.1, for that matter. It seems that there may be quite a number  
of corner cases in which either references are not counted correctly or  
hidden references are created that inhibit garbage collection. At least, I  
can't think of any connection between immediate part assignments and  
delayed OwnValue assignments to Temporary symbols that would explain the  
similarly erroneous behaviour observed in these two cases.

[toc] | [prev] | [standalone]


Back to top | Article view | comp.soft-sys.math.mathematica


csiph-web