Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #37411 > unrolled thread

Memory error with quadratic interpolation

Started byIsaac Won <winefrog@gmail.com>
First post2013-01-22 20:06 -0800
Last post2013-01-23 06:24 -0800
Articles 16 — 3 participants

Back to article view | Back to comp.lang.python

Contents

  Memory error with quadratic interpolation Isaac Won <winefrog@gmail.com> - 2013-01-22 20:06 -0800
    Re: Memory error with quadratic interpolation Ulrich Eckhardt <ulrich.eckhardt@dominolaser.com> - 2013-01-23 09:55 +0100
      Re: Memory error with quadratic interpolation Oscar Benjamin <oscar.j.benjamin@gmail.com> - 2013-01-23 10:08 +0000
        Re: Memory error with quadratic interpolation Isaac Won <winefrog@gmail.com> - 2013-01-23 06:26 -0800
        Re: Memory error with quadratic interpolation Isaac Won <winefrog@gmail.com> - 2013-01-23 06:26 -0800
        Re: Memory error with quadratic interpolation Isaac Won <winefrog@gmail.com> - 2013-01-23 06:28 -0800
          Re: Memory error with quadratic interpolation Oscar Benjamin <oscar.j.benjamin@gmail.com> - 2013-01-23 14:40 +0000
            Re: Memory error with quadratic interpolation Isaac Won <winefrog@gmail.com> - 2013-01-23 06:57 -0800
              Re: Memory error with quadratic interpolation Oscar Benjamin <oscar.j.benjamin@gmail.com> - 2013-01-23 16:51 +0000
                Re: Memory error with quadratic interpolation Isaac Won <winefrog@gmail.com> - 2013-01-23 09:33 -0800
                  Re: Memory error with quadratic interpolation Oscar Benjamin <oscar.j.benjamin@gmail.com> - 2013-01-24 10:30 +0000
                Re: Memory error with quadratic interpolation Isaac Won <winefrog@gmail.com> - 2013-01-23 09:33 -0800
            Re: Memory error with quadratic interpolation Isaac Won <winefrog@gmail.com> - 2013-01-23 06:57 -0800
        Re: Memory error with quadratic interpolation Isaac Won <winefrog@gmail.com> - 2013-01-23 06:28 -0800
      Re: Memory error with quadratic interpolation Isaac Won <winefrog@gmail.com> - 2013-01-23 06:47 -0800
    Re: Memory error with quadratic interpolation Isaac Won <winefrog@gmail.com> - 2013-01-23 06:24 -0800

#37411 — Memory error with quadratic interpolation

FromIsaac Won <winefrog@gmail.com>
Date2013-01-22 20:06 -0800
SubjectMemory error with quadratic interpolation
Message-ID<0909ee56-c304-4b9d-8be9-97c7a3261d27@googlegroups.com>
Hi all,

I have tried to use different interpolation methods with Scipy. My code seems just fine with linear interpolation, but shows memory error with quadratic. I am a novice for python. I will appreciate any help.

#code
f = open(filin, "r")
for columns in ( raw.strip().split() for raw in f ):
        a.append(columns[5])
        x = np.array(a, float)


not_nan = np.logical_not(np.isnan(x))
indices = np.arange(len(x))
interp = interp1d(indices[not_nan], x[not_nan], kind = 'quadratic')

p = interp(indices)

------------------------------------------------------------------------
The number of data is 31747.

Thank you,

Isaac

[toc] | [next] | [standalone]

#37428

FromUlrich Eckhardt <ulrich.eckhardt@dominolaser.com>
Date2013-01-23 09:55 +0100
Message-ID<j4l4t9-skt.ln1@satorlaser.homedns.org>
In reply to#37411
Am 23.01.2013 05:06, schrieb Isaac Won:
> I have tried to use different interpolation methods with Scipy. My
> code seems just fine with linear interpolation, but shows memory
> error with quadratic. I am a novice for python. I will appreciate any
> help.
 >
> #code
> f = open(filin, "r")

Check out the "with open(...) as f" syntax.


> for columns in ( raw.strip().split() for raw in f ):

For the record, this first builds a sequence and then iterates over that 
sequence. This is not very memory-efficient, try this instead:

    for line in f:
        columns = line.strip().split()


Concerning the rest of your problems, there is lots of code and the 
datafile missing. However, there is also too much of it, try replacing 
the file with generated data and remove everything from the code that is 
not absolutely necessary.

Good luck!

Uli

[toc] | [prev] | [next] | [standalone]

#37439

FromOscar Benjamin <oscar.j.benjamin@gmail.com>
Date2013-01-23 10:08 +0000
Message-ID<mailman.880.1358935694.2939.python-list@python.org>
In reply to#37428
On 23 January 2013 08:55, Ulrich Eckhardt
<ulrich.eckhardt@dominolaser.com> wrote:
> Am 23.01.2013 05:06, schrieb Isaac Won:
>
>> I have tried to use different interpolation methods with Scipy. My
>> code seems just fine with linear interpolation, but shows memory
>> error with quadratic. I am a novice for python. I will appreciate any
>> help.
>
[SNIP]
>
>
> Concerning the rest of your problems, there is lots of code and the datafile
> missing. However, there is also too much of it, try replacing the file with
> generated data and remove everything from the code that is not absolutely
> necessary.

Also please copy paste the actual error message rather than paraphrasing it.


Oscar

[toc] | [prev] | [next] | [standalone]

#37470

FromIsaac Won <winefrog@gmail.com>
Date2013-01-23 06:26 -0800
Message-ID<369aff35-d655-4706-bfae-3a9a0535b923@googlegroups.com>
In reply to#37439
On Wednesday, January 23, 2013 4:08:13 AM UTC-6, Oscar Benjamin wrote:
> On 23 January 2013 08:55, Ulrich Eckhardt
> 
> <ulrich.eckhardt@dominolaser.com> wrote:
> 
> > Am 23.01.2013 05:06, schrieb Isaac Won:
> 
> >
> 
> >> I have tried to use different interpolation methods with Scipy. My
> 
> >> code seems just fine with linear interpolation, but shows memory
> 
> >> error with quadratic. I am a novice for python. I will appreciate any
> 
> >> help.
> 
> >
> 
> [SNIP]
> 
> >
> 
> >
> 
> > Concerning the rest of your problems, there is lots of code and the datafile
> 
> > missing. However, there is also too much of it, try replacing the file with
> 
> > generated data and remove everything from the code that is not absolutely
> 
> > necessary.
> 
> 
> 
> Also please copy paste the actual error message rather than paraphrasing it.
> 
> 
> 
> 
> 
> Oscar

I really appreciate to both Ulich and Oscar. 

To Oscar 
My actual error message is: 
File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 311, in __init__
     self._spline = splmake(x,oriented_y,order=order) 
  File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 809, in splmake
     coefs = func(xk, yk, order, conds, B) 
  File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 530, in _find_smoothest
     u,s,vh = np.dual.svd(B) 
  File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/linalg/decomp_svd.py", line 91, in svd
     full_matrices=full_matrices, overwrite_a = overwrite_a) 
MemoryError 
-------------------------------------------------------------------------- 
Thank you, 

Isaac

[toc] | [prev] | [next] | [standalone]

#37471

FromIsaac Won <winefrog@gmail.com>
Date2013-01-23 06:26 -0800
Message-ID<mailman.896.1358951194.2939.python-list@python.org>
In reply to#37439
On Wednesday, January 23, 2013 4:08:13 AM UTC-6, Oscar Benjamin wrote:
> On 23 January 2013 08:55, Ulrich Eckhardt
> 
> <ulrich.eckhardt@dominolaser.com> wrote:
> 
> > Am 23.01.2013 05:06, schrieb Isaac Won:
> 
> >
> 
> >> I have tried to use different interpolation methods with Scipy. My
> 
> >> code seems just fine with linear interpolation, but shows memory
> 
> >> error with quadratic. I am a novice for python. I will appreciate any
> 
> >> help.
> 
> >
> 
> [SNIP]
> 
> >
> 
> >
> 
> > Concerning the rest of your problems, there is lots of code and the datafile
> 
> > missing. However, there is also too much of it, try replacing the file with
> 
> > generated data and remove everything from the code that is not absolutely
> 
> > necessary.
> 
> 
> 
> Also please copy paste the actual error message rather than paraphrasing it.
> 
> 
> 
> 
> 
> Oscar

I really appreciate to both Ulich and Oscar. 

To Oscar 
My actual error message is: 
File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 311, in __init__
     self._spline = splmake(x,oriented_y,order=order) 
  File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 809, in splmake
     coefs = func(xk, yk, order, conds, B) 
  File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 530, in _find_smoothest
     u,s,vh = np.dual.svd(B) 
  File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/linalg/decomp_svd.py", line 91, in svd
     full_matrices=full_matrices, overwrite_a = overwrite_a) 
MemoryError 
-------------------------------------------------------------------------- 
Thank you, 

Isaac

[toc] | [prev] | [next] | [standalone]

#37472

FromIsaac Won <winefrog@gmail.com>
Date2013-01-23 06:28 -0800
Message-ID<8daf86cf-386c-44ae-83d5-25fb04572c00@googlegroups.com>
In reply to#37439
On Wednesday, January 23, 2013 4:08:13 AM UTC-6, Oscar Benjamin wrote:
> On 23 January 2013 08:55, Ulrich Eckhardt
> 
> 
> 
> > Am 23.01.2013 05:06, schrieb Isaac Won:
> 
> >
> 
> >> I have tried to use different interpolation methods with Scipy. My
> 
> >> code seems just fine with linear interpolation, but shows memory
> 
> >> error with quadratic. I am a novice for python. I will appreciate any
> 
> >> help.
> 
> >
> 
> [SNIP]
> 
> >
> 
> >
> 
> > Concerning the rest of your problems, there is lots of code and the datafile
> 
> > missing. However, there is also too much of it, try replacing the file with
> 
> > generated data and remove everything from the code that is not absolutely
> 
> > necessary.
> 
> 
> 
> Also please copy paste the actual error message rather than paraphrasing it.
> 
> 
> 
> 
> 
> Oscar

I really appreciate to both Ulich and Oscar. 

To Oscar 
My actual error message is: 
File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 311, in __init__
     self._spline = splmake(x,oriented_y,order=order) 
  File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 809, in splmake
     coefs = func(xk, yk, order, conds, B) 
  File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 530, in _find_smoothest
     u,s,vh = np.dual.svd(B) 
  File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/linalg/decomp_svd.py", line 91, in svd
     full_matrices=full_matrices, overwrite_a = overwrite_a) 
MemoryError 
-------------------------------------------------------------------------- 
Thank you, 

Hoonill

[toc] | [prev] | [next] | [standalone]

#37474

FromOscar Benjamin <oscar.j.benjamin@gmail.com>
Date2013-01-23 14:40 +0000
Message-ID<mailman.898.1358952056.2939.python-list@python.org>
In reply to#37472
On 23 January 2013 14:28, Isaac Won <winefrog@gmail.com> wrote:
> On Wednesday, January 23, 2013 4:08:13 AM UTC-6, Oscar Benjamin wrote:
>
> To Oscar
> My actual error message is:
> File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 311, in __init__
>      self._spline = splmake(x,oriented_y,order=order)
>   File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 809, in splmake
>      coefs = func(xk, yk, order, conds, B)
>   File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 530, in _find_smoothest
>      u,s,vh = np.dual.svd(B)
>   File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/linalg/decomp_svd.py", line 91, in svd
>      full_matrices=full_matrices, overwrite_a = overwrite_a)
> MemoryError

Are you sure that's the *whole* error message? The traceback only
refers to the scipy modules. I can't see the line from your code that
is generating the error.


Oscar

[toc] | [prev] | [next] | [standalone]

#37478

FromIsaac Won <winefrog@gmail.com>
Date2013-01-23 06:57 -0800
Message-ID<a5662c8c-67c3-47e6-a8b7-2eddbc272657@googlegroups.com>
In reply to#37474
On Wednesday, January 23, 2013 8:40:54 AM UTC-6, Oscar Benjamin wrote:
> On 23 January 2013 14:28, Isaac Won <winefrog@gmail.com> wrote:
> 
> > On Wednesday, January 23, 2013 4:08:13 AM UTC-6, Oscar Benjamin wrote:
> 
> >
> 
> > To Oscar
> 
> > My actual error message is:
> 
> > File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 311, in __init__
> 
> >      self._spline = splmake(x,oriented_y,order=order)
> 
> >   File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 809, in splmake
> 
> >      coefs = func(xk, yk, order, conds, B)
> 
> >   File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 530, in _find_smoothest
> 
> >      u,s,vh = np.dual.svd(B)
> 
> >   File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/linalg/decomp_svd.py", line 91, in svd
> 
> >      full_matrices=full_matrices, overwrite_a = overwrite_a)
> 
> > MemoryError
> 
> 
> 
> Are you sure that's the *whole* error message? The traceback only
> 
> refers to the scipy modules. I can't see the line from your code that
> 
> is generating the error.
> 
> 
> 
> 
> 
> Oscar

Dear Oscar,

Following is full error message after I adjusted following Ulich's advice:

interp = interp1d(indices[not_nan], x[not_nan], kind = 'quadratic') 
File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 311, in __init__
     self._spline = splmake(x,oriented_y,order=order) 
  File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 809, in splmake
     coefs = func(xk, yk, order, conds, B) 
  File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 530, in _find_smoothest
     u,s,vh = np.dual.svd(B) 
  File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/linalg/decomp_svd.py", line 91, in svd
     full_matrices=full_matrices, overwrite_a = overwrite_a) 
MemoryError 
----------------------------------------------------------------------
Thank you,

Hoonill

[toc] | [prev] | [next] | [standalone]

#37491

FromOscar Benjamin <oscar.j.benjamin@gmail.com>
Date2013-01-23 16:51 +0000
Message-ID<mailman.910.1358959906.2939.python-list@python.org>
In reply to#37478
On 23 January 2013 14:57, Isaac Won <winefrog@gmail.com> wrote:
> On Wednesday, January 23, 2013 8:40:54 AM UTC-6, Oscar Benjamin wrote:
>> On 23 January 2013 14:28, Isaac Won <winefrog@gmail.com> wrote:
>>
[SNIP]
>
> Following is full error message after I adjusted following Ulich's advice:
>
> interp = interp1d(indices[not_nan], x[not_nan], kind = 'quadratic')
> File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 311, in __init__
>      self._spline = splmake(x,oriented_y,order=order)
>   File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 809, in splmake
>      coefs = func(xk, yk, order, conds, B)
>   File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 530, in _find_smoothest
>      u,s,vh = np.dual.svd(B)
>   File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/linalg/decomp_svd.py", line 91, in svd
>      full_matrices=full_matrices, overwrite_a = overwrite_a)
> MemoryError

Where is the new code? You should show full working code (with the
import statements) and the full error that is generated by exactly
that code. If possible you should also write code that someone else
could run even without having access to your data files. If you did
that in your first post, you'd probably have an answer to your problem
by now.

Here is a version of your code that many people on this list can test
straight away:

import numpy as np
from scipy.interpolate import interp1d
x = np.array(31747 * [0.0], float)
indices = np.arange(len(x))
interp = interp1d(indices, x, kind='quadratic')

Running this gives the following error:

~$ python tmp.py
Traceback (most recent call last):
  File "tmp.py", line 5, in <module>
    interp = interp1d(indices, x, kind='quadratic')
  File "/usr/lib/python2.7/dist-packages/scipy/interpolate/interpolate.py",
line 308, in __init__
    self._spline = splmake(x,oriented_y,order=order)
  File "/usr/lib/python2.7/dist-packages/scipy/interpolate/interpolate.py",
line 805, in splmake
    B = _fitpack._bsplmat(order, xk)
MemoryError

Unless I've misunderstood how this function is supposed to be used, it
just doesn't really seem to work for arrays of much more than a few
hundred elements.


Oscar

[toc] | [prev] | [next] | [standalone]

#37495

FromIsaac Won <winefrog@gmail.com>
Date2013-01-23 09:33 -0800
Message-ID<94ef456a-af84-48fe-8007-df42c4a625c4@googlegroups.com>
In reply to#37491
On Wednesday, January 23, 2013 10:51:43 AM UTC-6, Oscar Benjamin wrote:
> On 23 January 2013 14:57, Isaac Won <winefrog@gmail.com> wrote:
> 
> > On Wednesday, January 23, 2013 8:40:54 AM UTC-6, Oscar Benjamin wrote:
> 
> >> On 23 January 2013 14:28, Isaac Won <winefrog@gmail.com> wrote:
> 
> >>
> 
> [SNIP]
> 
> >
> 
> > Following is full error message after I adjusted following Ulich's advice:
> 
> >
> 
> > interp = interp1d(indices[not_nan], x[not_nan], kind = 'quadratic')
> 
> > File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 311, in __init__
> 
> >      self._spline = splmake(x,oriented_y,order=order)
> 
> >   File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 809, in splmake
> 
> >      coefs = func(xk, yk, order, conds, B)
> 
> >   File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 530, in _find_smoothest
> 
> >      u,s,vh = np.dual.svd(B)
> 
> >   File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/linalg/decomp_svd.py", line 91, in svd
> 
> >      full_matrices=full_matrices, overwrite_a = overwrite_a)
> 
> > MemoryError
> 
> 
> 
> Where is the new code? You should show full working code (with the
> 
> import statements) and the full error that is generated by exactly
> 
> that code. If possible you should also write code that someone else
> 
> could run even without having access to your data files. If you did
> 
> that in your first post, you'd probably have an answer to your problem
> 
> by now.
> 
> 
> 
> Here is a version of your code that many people on this list can test
> 
> straight away:
> 
> 
> 
> import numpy as np
> 
> from scipy.interpolate import interp1d
> 
> x = np.array(31747 * [0.0], float)
> 
> indices = np.arange(len(x))
> 
> interp = interp1d(indices, x, kind='quadratic')
> 
> 
> 
> Running this gives the following error:
> 
> 
> 
> ~$ python tmp.py
> 
> Traceback (most recent call last):
> 
>   File "tmp.py", line 5, in <module>
> 
>     interp = interp1d(indices, x, kind='quadratic')
> 
>   File "/usr/lib/python2.7/dist-packages/scipy/interpolate/interpolate.py",
> 
> line 308, in __init__
> 
>     self._spline = splmake(x,oriented_y,order=order)
> 
>   File "/usr/lib/python2.7/dist-packages/scipy/interpolate/interpolate.py",
> 
> line 805, in splmake
> 
>     B = _fitpack._bsplmat(order, xk)
> 
> MemoryError
> 
> 
> 
> Unless I've misunderstood how this function is supposed to be used, it
> 
> just doesn't really seem to work for arrays of much more than a few
> 
> hundred elements.
> 
> 
> 
> 
> 
> Oscar

Thank you Oscar for your help and advice.

I agree with you. So, I tried to find the way to solve this problem.

My full code adjusted is:
from scipy.interpolate import interp1d

import numpy as np
import matplotlib.pyplot as plt



with open(filin, "r") as f:

        for line in f:
                columns = line.strip().split()

                a.append(columns[5])
                x = np.array(a, float)


not_nan = np.logical_not(np.isnan(x))
indices = np.arange(len(x))
interp = interp1d(indices[not_nan], x[not_nan], kind = 'quadratic')

p = interp(indices)




k = np.arange(31747)

plt.subplot(211)
plt.plot(k, p)
plt.xlabel('Quadratic interpolation')
plt.subplot(212)
plt.plot(k, x)

plt.show()
-----------------------------------------------------------------
Whole error message was:

Traceback (most recent call last):
  File "QI1.py", line 22, in <module>
    interp = interp1d(indices[not_nan], x[not_nan], kind = 'quadratic')
  File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 311, in __init__
    self._spline = splmake(x,oriented_y,order=order)
  File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 809, in splmake
    coefs = func(xk, yk, order, conds, B)
  File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 530, in _find_smoothest
    u,s,vh = np.dual.svd(B)
  File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/linalg/decomp_svd.py", line 91, in svd
    full_matrices=full_matrices, overwrite_a = overwrite_a)
MemoryError
----------------------------------------------------------------------
Thank you again Oscar,

Isaac

[toc] | [prev] | [next] | [standalone]

#37550

FromOscar Benjamin <oscar.j.benjamin@gmail.com>
Date2013-01-24 10:30 +0000
Message-ID<mailman.952.1359023458.2939.python-list@python.org>
In reply to#37495
On 23 January 2013 17:33, Isaac Won <winefrog@gmail.com> wrote:
> On Wednesday, January 23, 2013 10:51:43 AM UTC-6, Oscar Benjamin wrote:
>> On 23 January 2013 14:57, Isaac Won <winefrog@gmail.com> wrote:
>>
>> > On Wednesday, January 23, 2013 8:40:54 AM UTC-6, Oscar Benjamin wrote:
>>
>> Unless I've misunderstood how this function is supposed to be used, it
>> just doesn't really seem to work for arrays of much more than a few
>> hundred elements.
>>

The solution is to use UnivariateSpline. I don't know what the
difference is but it works where the other fails:

import numpy as np
from scipy.interpolate import UnivariateSpline
x = np.array(10000 * [0.0], float)
indices = np.arange(len(x))
interp = UnivariateSpline(indices, x, k=2)
print(interp(1.5))


Oscar

[toc] | [prev] | [next] | [standalone]

#37496

FromIsaac Won <winefrog@gmail.com>
Date2013-01-23 09:33 -0800
Message-ID<mailman.914.1358962421.2939.python-list@python.org>
In reply to#37491
On Wednesday, January 23, 2013 10:51:43 AM UTC-6, Oscar Benjamin wrote:
> On 23 January 2013 14:57, Isaac Won <winefrog@gmail.com> wrote:
> 
> > On Wednesday, January 23, 2013 8:40:54 AM UTC-6, Oscar Benjamin wrote:
> 
> >> On 23 January 2013 14:28, Isaac Won <winefrog@gmail.com> wrote:
> 
> >>
> 
> [SNIP]
> 
> >
> 
> > Following is full error message after I adjusted following Ulich's advice:
> 
> >
> 
> > interp = interp1d(indices[not_nan], x[not_nan], kind = 'quadratic')
> 
> > File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 311, in __init__
> 
> >      self._spline = splmake(x,oriented_y,order=order)
> 
> >   File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 809, in splmake
> 
> >      coefs = func(xk, yk, order, conds, B)
> 
> >   File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 530, in _find_smoothest
> 
> >      u,s,vh = np.dual.svd(B)
> 
> >   File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/linalg/decomp_svd.py", line 91, in svd
> 
> >      full_matrices=full_matrices, overwrite_a = overwrite_a)
> 
> > MemoryError
> 
> 
> 
> Where is the new code? You should show full working code (with the
> 
> import statements) and the full error that is generated by exactly
> 
> that code. If possible you should also write code that someone else
> 
> could run even without having access to your data files. If you did
> 
> that in your first post, you'd probably have an answer to your problem
> 
> by now.
> 
> 
> 
> Here is a version of your code that many people on this list can test
> 
> straight away:
> 
> 
> 
> import numpy as np
> 
> from scipy.interpolate import interp1d
> 
> x = np.array(31747 * [0.0], float)
> 
> indices = np.arange(len(x))
> 
> interp = interp1d(indices, x, kind='quadratic')
> 
> 
> 
> Running this gives the following error:
> 
> 
> 
> ~$ python tmp.py
> 
> Traceback (most recent call last):
> 
>   File "tmp.py", line 5, in <module>
> 
>     interp = interp1d(indices, x, kind='quadratic')
> 
>   File "/usr/lib/python2.7/dist-packages/scipy/interpolate/interpolate.py",
> 
> line 308, in __init__
> 
>     self._spline = splmake(x,oriented_y,order=order)
> 
>   File "/usr/lib/python2.7/dist-packages/scipy/interpolate/interpolate.py",
> 
> line 805, in splmake
> 
>     B = _fitpack._bsplmat(order, xk)
> 
> MemoryError
> 
> 
> 
> Unless I've misunderstood how this function is supposed to be used, it
> 
> just doesn't really seem to work for arrays of much more than a few
> 
> hundred elements.
> 
> 
> 
> 
> 
> Oscar

Thank you Oscar for your help and advice.

I agree with you. So, I tried to find the way to solve this problem.

My full code adjusted is:
from scipy.interpolate import interp1d

import numpy as np
import matplotlib.pyplot as plt



with open(filin, "r") as f:

        for line in f:
                columns = line.strip().split()

                a.append(columns[5])
                x = np.array(a, float)


not_nan = np.logical_not(np.isnan(x))
indices = np.arange(len(x))
interp = interp1d(indices[not_nan], x[not_nan], kind = 'quadratic')

p = interp(indices)




k = np.arange(31747)

plt.subplot(211)
plt.plot(k, p)
plt.xlabel('Quadratic interpolation')
plt.subplot(212)
plt.plot(k, x)

plt.show()
-----------------------------------------------------------------
Whole error message was:

Traceback (most recent call last):
  File "QI1.py", line 22, in <module>
    interp = interp1d(indices[not_nan], x[not_nan], kind = 'quadratic')
  File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 311, in __init__
    self._spline = splmake(x,oriented_y,order=order)
  File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 809, in splmake
    coefs = func(xk, yk, order, conds, B)
  File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 530, in _find_smoothest
    u,s,vh = np.dual.svd(B)
  File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/linalg/decomp_svd.py", line 91, in svd
    full_matrices=full_matrices, overwrite_a = overwrite_a)
MemoryError
----------------------------------------------------------------------
Thank you again Oscar,

Isaac

[toc] | [prev] | [next] | [standalone]

#37479

FromIsaac Won <winefrog@gmail.com>
Date2013-01-23 06:57 -0800
Message-ID<mailman.901.1358953068.2939.python-list@python.org>
In reply to#37474
On Wednesday, January 23, 2013 8:40:54 AM UTC-6, Oscar Benjamin wrote:
> On 23 January 2013 14:28, Isaac Won <winefrog@gmail.com> wrote:
> 
> > On Wednesday, January 23, 2013 4:08:13 AM UTC-6, Oscar Benjamin wrote:
> 
> >
> 
> > To Oscar
> 
> > My actual error message is:
> 
> > File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 311, in __init__
> 
> >      self._spline = splmake(x,oriented_y,order=order)
> 
> >   File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 809, in splmake
> 
> >      coefs = func(xk, yk, order, conds, B)
> 
> >   File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 530, in _find_smoothest
> 
> >      u,s,vh = np.dual.svd(B)
> 
> >   File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/linalg/decomp_svd.py", line 91, in svd
> 
> >      full_matrices=full_matrices, overwrite_a = overwrite_a)
> 
> > MemoryError
> 
> 
> 
> Are you sure that's the *whole* error message? The traceback only
> 
> refers to the scipy modules. I can't see the line from your code that
> 
> is generating the error.
> 
> 
> 
> 
> 
> Oscar

Dear Oscar,

Following is full error message after I adjusted following Ulich's advice:

interp = interp1d(indices[not_nan], x[not_nan], kind = 'quadratic') 
File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 311, in __init__
     self._spline = splmake(x,oriented_y,order=order) 
  File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 809, in splmake
     coefs = func(xk, yk, order, conds, B) 
  File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 530, in _find_smoothest
     u,s,vh = np.dual.svd(B) 
  File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/linalg/decomp_svd.py", line 91, in svd
     full_matrices=full_matrices, overwrite_a = overwrite_a) 
MemoryError 
----------------------------------------------------------------------
Thank you,

Hoonill

[toc] | [prev] | [next] | [standalone]

#37473

FromIsaac Won <winefrog@gmail.com>
Date2013-01-23 06:28 -0800
Message-ID<mailman.897.1358951293.2939.python-list@python.org>
In reply to#37439
On Wednesday, January 23, 2013 4:08:13 AM UTC-6, Oscar Benjamin wrote:
> On 23 January 2013 08:55, Ulrich Eckhardt
> 
> 
> 
> > Am 23.01.2013 05:06, schrieb Isaac Won:
> 
> >
> 
> >> I have tried to use different interpolation methods with Scipy. My
> 
> >> code seems just fine with linear interpolation, but shows memory
> 
> >> error with quadratic. I am a novice for python. I will appreciate any
> 
> >> help.
> 
> >
> 
> [SNIP]
> 
> >
> 
> >
> 
> > Concerning the rest of your problems, there is lots of code and the datafile
> 
> > missing. However, there is also too much of it, try replacing the file with
> 
> > generated data and remove everything from the code that is not absolutely
> 
> > necessary.
> 
> 
> 
> Also please copy paste the actual error message rather than paraphrasing it.
> 
> 
> 
> 
> 
> Oscar

I really appreciate to both Ulich and Oscar. 

To Oscar 
My actual error message is: 
File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 311, in __init__
     self._spline = splmake(x,oriented_y,order=order) 
  File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 809, in splmake
     coefs = func(xk, yk, order, conds, B) 
  File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 530, in _find_smoothest
     u,s,vh = np.dual.svd(B) 
  File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/linalg/decomp_svd.py", line 91, in svd
     full_matrices=full_matrices, overwrite_a = overwrite_a) 
MemoryError 
-------------------------------------------------------------------------- 
Thank you, 

Hoonill

[toc] | [prev] | [next] | [standalone]

#37475

FromIsaac Won <winefrog@gmail.com>
Date2013-01-23 06:47 -0800
Message-ID<1daeb571-3769-455a-b2d6-e784d816db58@googlegroups.com>
In reply to#37428
On Wednesday, January 23, 2013 2:55:14 AM UTC-6, Ulrich Eckhardt wrote:
> Am 23.01.2013 05:06, schrieb Isaac Won:
> 
> > I have tried to use different interpolation methods with Scipy. My
> 
> > code seems just fine with linear interpolation, but shows memory
> 
> > error with quadratic. I am a novice for python. I will appreciate any
> 
> > help.
> 
>  >
> 
> > #code
> 
> > f = open(filin, "r")
> 
> 
> 
> Check out the "with open(...) as f" syntax.
> 
> 
> 
> 
> 
> > for columns in ( raw.strip().split() for raw in f ):
> 
> 
> 
> For the record, this first builds a sequence and then iterates over that 
> 
> sequence. This is not very memory-efficient, try this instead:
> 
> 
> 
>     for line in f:
> 
>         columns = line.strip().split()
> 
> 
> 
> 
> 
> Concerning the rest of your problems, there is lots of code and the 
> 
> datafile missing. However, there is also too much of it, try replacing 
> 
> the file with generated data and remove everything from the code that is 
> 
> not absolutely necessary.
> 
> 
> 
> Good luck!
> 
> 
> 
> Uli

Hi Ulich,

I tried to change the code following your advice, but it doesn't seem to work still.

My adjusted code is:

a = []

with open(filin, "r") as f:

        for line in f:
                columns = line.strip().split()

                a.append(columns[5])
                x = np.array(a, float)


not_nan = np.logical_not(np.isnan(x))
indices = np.arange(len(x))
interp = interp1d(indices[not_nan], x[not_nan], kind = 'quadratic')
p = interp(indices)
---------------------------------------------------------------------
And full error message is:
   interp = interp1d(indices[not_nan], x[not_nan], kind = 'quadratic')
  File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 311, in __init__
    self._spline = splmake(x,oriented_y,order=order)
  File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 809, in splmake
    coefs = func(xk, yk, order, conds, B)
  File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 530, in _find_smoothest
    u,s,vh = np.dual.svd(B)
  File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/linalg/decomp_svd.py", line 91, in svd
    full_matrices=full_matrices, overwrite_a = overwrite_a)
MemoryError
-----------------------------------------------------------------------
Could you give me some advice for this situation?

Thank you always,

Isaac

[toc] | [prev] | [next] | [standalone]

#37468

FromIsaac Won <winefrog@gmail.com>
Date2013-01-23 06:24 -0800
Message-ID<9686dfe5-3229-4040-8e16-ac0d039d1b3f@googlegroups.com>
In reply to#37411
On Tuesday, January 22, 2013 10:06:41 PM UTC-6, Isaac Won wrote:
> Hi all,
> 
> 
> 
> I have tried to use different interpolation methods with Scipy. My code seems just fine with linear interpolation, but shows memory error with quadratic. I am a novice for python. I will appreciate any help.
> 
> 
> 
> #code
> 
> f = open(filin, "r")
> 
> for columns in ( raw.strip().split() for raw in f ):
> 
>         a.append(columns[5])
> 
>         x = np.array(a, float)
> 
> 
> 
> 
> 
> not_nan = np.logical_not(np.isnan(x))
> 
> indices = np.arange(len(x))
> 
> interp = interp1d(indices[not_nan], x[not_nan], kind = 'quadratic')
> 
> 
> 
> p = interp(indices)
> 
> 
> 
> ------------------------------------------------------------------------
> 
> The number of data is 31747.
> 
> 
> 
> Thank you,
> 
> 
> 
> Isaac

I really appreciate to both Ulich and Oscar.

To Oscar
My actual error message is:
File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 311, in __init__
    self._spline = splmake(x,oriented_y,order=order)
  File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 809, in splmake
    coefs = func(xk, yk, order, conds, B)
  File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 530, in _find_smoothest
    u,s,vh = np.dual.svd(B)
  File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/linalg/decomp_svd.py", line 91, in svd
    full_matrices=full_matrices, overwrite_a = overwrite_a)
MemoryError
--------------------------------------------------------------------------
Thank you,

Hoonill

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web