Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #95342 > unrolled thread

problem with netCDF4 OpenDAP

Back to article view | Back to comp.lang.python

Contents

  problem with netCDF4 OpenDAP Tom P <werotizy@freent.dd> - 2015-08-13 12:32 +0200
    Re: problem with netCDF4 OpenDAP Tom P <werotizy@freent.dd> - 2015-08-14 09:18 +0200
      Re: problem with netCDF4 OpenDAP Jason Swails <jason.swails@gmail.com> - 2015-08-14 09:15 -0400
        Re: problem with netCDF4 OpenDAP Tom P <werotizy@freent.dd> - 2015-08-14 17:26 +0200
          Re: problem with netCDF4 OpenDAP Michael Ströder <michael@stroeder.com> - 2015-08-15 02:02 +0200

#95342 — problem with netCDF4 OpenDAP

I'm having a problem trying to access OpenDAP files using netCDF4.
The netCDF4 is installed from the Anaconda package. According to their 
changelog, openDAP is supposed to be supported.

netCDF4.__version__
Out[7]:
'1.1.8'

Here's some code:

url = 
'http://www1.ncdc.noaa.gov/pub/data/cmb/ersst/v3b/netcdf/ersst.201507.nc'
nc = netCDF4.Dataset(url)

I get the error -
netCDF4/_netCDF4.pyx in netCDF4._netCDF4.Dataset.__init__ 
(netCDF4/_netCDF4.c:9551)()

RuntimeError: NetCDF: file not found


However if I download the same file, it works -
url = '/home/tom/Downloads/ersst.201507.nc'
nc = netCDF4.Dataset(url)
print nc
  . . . .

Is it something I'm doing wrong?

[toc] | [next] | [standalone]

#95373

On 08/13/2015 05:55 PM, Jason Swails wrote:
>
>
> On Thu, Aug 13, 2015 at 6:32 AM, Tom P <werotizy@freent.dd
> <mailto:werotizy@freent.dd>> wrote:
>
>     I'm having a problem trying to access OpenDAP files using netCDF4.
>     The netCDF4 is installed from the Anaconda package. According to
>     their changelog, openDAP is supposed to be supported.
>
>     netCDF4.__version__
>     Out[7]:
>     '1.1.8'
>
>     Here's some code:
>
>     url =
>     'http://www1.ncdc.noaa.gov/pub/data/cmb/ersst/v3b/netcdf/ersst.201507.nc'
>     nc = netCDF4.Dataset(url)
>
>     I get the error -
>     netCDF4/_netCDF4.pyx in netCDF4._netCDF4.Dataset.__init__
>     (netCDF4/_netCDF4.c:9551)()
>
>     RuntimeError: NetCDF: file not found
>
>
>     However if I download the same file, it works -
>     url = '/home/tom/Downloads/ersst.201507.nc <http://ersst.201507.nc>'
>     nc = netCDF4.Dataset(url)
>     print nc
>       . . . .
>
>     Is it something I'm doing wrong?
>
>
> ​Yes.  URLs are not files and cannot be opened like normal files.
>   netCDF4 *requires* a local file as far as I can tell.
>
> All the best,
> Jason
>
> --
> Jason M. Swails
> BioMaPS,
> Rutgers University
> Postdoctoral Researcher

Thanks for the reply but that is not what the documentation says.

http://unidata.github.io/netcdf4-python/#section8
"Remote OPeNDAP-hosted datasets can be accessed for reading over http if 
a URL is provided to the netCDF4.Dataset constructor instead of a 
filename. However, this requires that the netCDF library be built with 
OPenDAP support, via the --enable-dap configure option (added in version 
4.0.1)."

and for the Anaconda package -
http://docs.continuum.io/anaconda/changelog
"2013-05-08: 1.5.0:
Highlights:
   updates to all important packages: python, numpy, scipy, ipython, 
matplotlib, pandas, cython
   added netCDF4 (with OpenDAP support) on Linux and MacOSX"

[toc] | [prev] | [next] | [standalone]

#95375

> On Aug 14, 2015, at 3:18 AM, Tom P <werotizy@freent.dd> wrote:
> 
> Thanks for the reply but that is not what the documentation says.
> 
> http://unidata.github.io/netcdf4-python/#section8
> "Remote OPeNDAP-hosted datasets can be accessed for reading over http if a URL is provided to the netCDF4.Dataset constructor instead of a filename. However, this requires that the netCDF library be built with OPenDAP support, via the --enable-dap configure option (added in version 4.0.1).”

​Huh, so it does.  Your error message says "file not found", though, which suggested to me that it's trying to interpret the NetCDF file as a local file instead of a URL.  Indeed, when I run that example, the traceback is more complete (the traceback you printed had omitted some information):

>>> netCDF4.Dataset('http://www1.ncdc.noaa.gov/pub/data/cmb/ersst/v3b/netcdf/ersst.201507.nc')
syntax error, unexpected WORD_WORD, expecting SCAN_ATTR or SCAN_DATASET or SCAN_ERROR
context: <!DOCTYPE^ HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"><html><head><title>404 Not Found</title></head><body><h1>Not Found</h1><p>The requested URL /pub/data/cmb/ersst/v3b/netcdf/ersst.201507.nc.dds was not found on this server.</p></body></html>
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "netCDF4/_netCDF4.pyx", line 1547, in netCDF4._netCDF4.Dataset.__init__ (netCDF4/_netCDF4.c:9551)
RuntimeError: NetCDF: file not found

So it’s clear that netCDF4 is at least *trying* to go online to look for the file, but it simply can’t find it.  Since the docs say it’s linking to libcurl, I tried using curl to download the file (curl -# http://www1.ncdc.noaa.gov/pub/data/cmb/ersst/v3b/netcdf/ersst.201507.nc > test.nc) and it worked fine.  What’s more, it *seems* like the file (/pub/.../ersst.201507.nc.dds) was decorated with the ‘.dds’ suffix for some reason (not sure if the server redirected the request there or not).  But this looks like a netCDF4 issue.  Perhaps you can go to their project page on Github and file an issue there -- they will be more likely to have your answer than people here.

HTH,
Jason

> 
> and for the Anaconda package -
> http://docs.continuum.io/anaconda/changelog
> "2013-05-08: 1.5.0:
> Highlights:
>  updates to all important packages: python, numpy, scipy, ipython, matplotlib, pandas, cython
>  added netCDF4 (with OpenDAP support) on Linux and MacOSX"
> 
> -- 
> https://mail.python.org/mailman/listinfo/python-list

--
Jason M. Swails
BioMaPS,
Rutgers University
Postdoctoral Researcher

[toc] | [prev] | [next] | [standalone]

#95376

On 08/14/2015 03:15 PM, Jason Swails wrote:
>
>> On Aug 14, 2015, at 3:18 AM, Tom P <werotizy@freent.dd> wrote:
>>
>> Thanks for the reply but that is not what the documentation says.
>>
>> http://unidata.github.io/netcdf4-python/#section8
>> "Remote OPeNDAP-hosted datasets can be accessed for reading over http if a URL is provided to the netCDF4.Dataset constructor instead of a filename. However, this requires that the netCDF library be built with OPenDAP support, via the --enable-dap configure option (added in version 4.0.1).”
>
> ​Huh, so it does.  Your error message says "file not found", though, which suggested to me that it's trying to interpret the NetCDF file as a local file instead of a URL.  Indeed, when I run that example, the traceback is more complete (the traceback you printed had omitted some information):
>
>>>> netCDF4.Dataset('http://www1.ncdc.noaa.gov/pub/data/cmb/ersst/v3b/netcdf/ersst.201507.nc')
> syntax error, unexpected WORD_WORD, expecting SCAN_ATTR or SCAN_DATASET or SCAN_ERROR
> context: <!DOCTYPE^ HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"><html><head><title>404 Not Found</title></head><body><h1>Not Found</h1><p>The requested URL /pub/data/cmb/ersst/v3b/netcdf/ersst.201507.nc.dds was not found on this server.</p></body></html>
> Traceback (most recent call last):
>    File "<stdin>", line 1, in <module>
>    File "netCDF4/_netCDF4.pyx", line 1547, in netCDF4._netCDF4.Dataset.__init__ (netCDF4/_netCDF4.c:9551)
> RuntimeError: NetCDF: file not found
>
> So it’s clear that netCDF4 is at least *trying* to go online to look for the file, but it simply can’t find it.  Since the docs say it’s linking to libcurl, I tried using curl to download the file (curl -# http://www1.ncdc.noaa.gov/pub/data/cmb/ersst/v3b/netcdf/ersst.201507.nc > test.nc) and it worked fine.  What’s more, it *seems* like the file (/pub/.../ersst.201507.nc.dds) was decorated with the ‘.dds’ suffix for some reason (not sure if the server redirected the request there or not).  But this looks like a netCDF4 issue.  Perhaps you can go to their project page on Github and file an issue there -- they will be more likely to have your answer than people here.
>
> HTH,
> Jason

Hi,
   yes the file does appear to be there, I can download it and I can 
open and read the URL using urllib. Since there are a whole bunch of 
files in the directory, I really need MFDataset, but according to the 
documentation that doesn't work with URLs. Maybe the solution really is 
to D/L them all into a temporary folder and use MFDataset.

>
>>
>> and for the Anaconda package -
>> http://docs.continuum.io/anaconda/changelog
>> "2013-05-08: 1.5.0:
>> Highlights:
>>   updates to all important packages: python, numpy, scipy, ipython, matplotlib, pandas, cython
>>   added netCDF4 (with OpenDAP support) on Linux and MacOSX"
>>
>> --
>> https://mail.python.org/mailman/listinfo/python-list
>
> --
> Jason M. Swails
> BioMaPS,
> Rutgers University
> Postdoctoral Researcher
>

[toc] | [prev] | [next] | [standalone]

#95385

Tom P wrote:
>   yes the file does appear to be there, I can download it and I can open and
> read the URL using urllib. Since there are a whole bunch of files in the
> directory, I really need MFDataset, but according to the documentation that
> doesn't work with URLs. Maybe the solution really is to D/L them all into a
> temporary folder and use MFDataset.

Not sure about the size and other aspects of your deployment. But the safest
way to backup an OpenLDAP database is to export it to a single LDIF file
because this can be done while slapd is running and it's guaranteed that the
LDIF contains only data of finished transactions.

Ciao, Michael.

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web