Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #100513 > unrolled thread

geostationary satellite data

Back to article view | Back to comp.lang.python

Contents

  geostationary satellite data jorge.conrado@cptec.inpe.br - 2015-12-16 13:19 -0200
    Re: geostationary satellite data Peter Pearson <pkpearson@nowhere.invalid> - 2015-12-16 17:04 +0000
      Re: geostationary satellite data Chris Angelico <rosuav@gmail.com> - 2015-12-17 04:08 +1100
        Re: geostationary satellite data Peter Pearson <pkpearson@nowhere.invalid> - 2015-12-16 17:30 +0000
          Re: geostationary satellite data Chris Angelico <rosuav@gmail.com> - 2015-12-17 04:37 +1100
            Re: geostationary satellite data Peter Pearson <pkpearson@nowhere.invalid> - 2015-12-16 17:53 +0000
          Re: geostationary satellite data Robin Koch <robin.koch@t-online.de> - 2015-12-16 19:57 +0100

#100513 — geostationary satellite data

Hi,


I dowmloaded some data from the Mirador NASA site:


http://mirador.gsfc.nasa.gov/cgi-bin/mirador/presentNavigation.pl?tree=project&dataset=Global-merged%20IR%20Brightness%20Temperature%20Data&project=TRMM&dataGroup=Ancillary&version=001&CGISESSID=97f4b9244878c87819b2a1144d31e270


Each data have the dimension:  9896 x 3298 byte.


I used to read the command :

f = open('merg_2015110500_4km-pixel', mode='rb')

image = f.read()


Please, what can I do to visualize this data.


Conrado

[toc] | [next] | [standalone]

#100526

On Wed, 16 Dec 2015 13:19:26 -0200, jorge.conrado@cptec.inpe.br wrote:
>
> I dowmloaded some data from the Mirador NASA site:
>
> http://mirador.gsfc.nasa.gov/cgi-bin/mirador/presentNavigation.pl?tree=project&dataset=Global-merged%20IR%20Brightness%20Temperature%20Data&project=TRMM&dataGroup=Ancillary&version=001&CGISESSID=97f4b9244878c87819b2a1144d31e270
>
> Each data have the dimension:  9896 x 3298 byte.
>
> I used to read the command :
>
> f = open('merg_2015110500_4km-pixel', mode='rb')
>
> image = f.read()
>
> Please, what can I do to visualize this data.

You provide a URL to a web page, but you're opening a file.  Can
you tell us what kind of file you downloaded?

I grabbed a random file from that site, and it arrived with the name
"merg_2015120123_4km-pixel.Z".  After I ran "uncompress" on it, it was
named "merg_2015120123_4km-pixel", which looks much like your filename,
so I guess this is what you've done.  When I dump it as bytes, I get
a lot of this:

00000100  ff ff c0 c0 c1 c2 c2 c2  c2 c3 c3 c3 c3 c4 c4 c3
00000110  c3 c3 c1 c1 c0 c0 ff ff  ff ff ff ff ff ff c1 c1
00000120  c3 c4 c4 c4 c3 c3 c3 c3  c1 ff ff ff ff ff ff ff
00000130  ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff
00000140  ff ff ff ff ff ff ff c0  c0 c1 c3 c3 c3 c3 c4 c4
00000150  c4 c4 c4 c3 c3 c3 c4 c4  c3 c3 c3 c3 c3 c3 c3 c3
00000160  c4 c4 c4 c4 c4 c5 c5 c6  c7 c7 c7 c7 c5 c5 c5 c6
00000170  c6 c6 c6 c6 c4 c4 c4 c4  c4 c3 c3 c3 c3 c3 c3 c3
00000180  c2 c2 c2 c3 c3 c3 c4 c4  c4 c4 c4 c4 c4 c5 c5 c6
00000190  c6 c6 c6 c6 c6 c7 c7 c7  c7 c7 c7 c7 c7 c7 c7 c7
000001a0  c7 c7 c7 c8 c8 c8 c8 c8  c9 c9 c9 c9 c9 c9 c9 c9

The file is 65274016 bytes long.  You claim the dimensions are
9896 x 3298, but that comes out to half that number (32637008), so I'll
bet the real dimensions are 9896 x 6596, with one byte per pixel.
I think this image format is called "raw".

I haven't the time and expertise to settle this for you, but
the solution is probably going to look something like this:

>>> rawdata = open("merg_2015120123_4km-pixel", "rb").read()
>>> from PIL import Image
>>> img = Image.fromstring("L", (9896, 6596), rawdata)
>>> img.save("temp.png")


-- 
To email me, substitute nowhere->runbox, invalid->com.

[toc] | [prev] | [next] | [standalone]

#100527

On Thu, Dec 17, 2015 at 4:04 AM, Peter Pearson
<pkpearson@nowhere.invalid> wrote:
> The file is 65274016 bytes long.  You claim the dimensions are
> 9896 x 3298, but that comes out to half that number (32637008), so I'll
> bet the real dimensions are 9896 x 6596, with one byte per pixel.
> I think this image format is called "raw".

It could be 16 bits per pixel. Without knowing a lot more about the
source of the image and its format, it's hard to say with any
certainty.

ChrisA

[toc] | [prev] | [next] | [standalone]

#100530

On Thu, 17 Dec 2015 04:08:02 +1100, Chris Angelico <rosuav@gmail.com> wrote:
> On Thu, Dec 17, 2015 at 4:04 AM, Peter Pearson
><pkpearson@nowhere.invalid> wrote:
>> The file is 65274016 bytes long.  You claim the dimensions are
>> 9896 x 3298, but that comes out to half that number (32637008), so I'll
>> bet the real dimensions are 9896 x 6596, with one byte per pixel.
>> I think this image format is called "raw".
>
> It could be 16 bits per pixel. Without knowing a lot more about the
> source of the image and its format, it's hard to say with any
> certainty.

Agreed.  It's annoying when an agency goes to the trouble of making
huge datasets available online, but fails to identify the format.

But the 16-bits-per-pixel hypothesis is unlikely, given that each
byte tends to echo its predecessor:

0000130 ffff ffff ffff ffff ffff ffff ffff ffff
0000140 ffff ffff ffff c0ff c1c0 c3c3 c3c3 c4c4
0000150 c4c4 c3c4 c3c3 c4c4 c3c3 c3c3 c3c3 c3c3
0000160 c4c4 c4c4 c5c4 c6c5 c7c7 c7c7 c5c5 c6c5

When you decompose this data file as a one-byte-per-pixel, 9896 x 6596
image, the resulting image shows two nearly identical strips, one above
the other.  That suggests interlacing, except that the top strip has
some "bites" missing that aren't missing from the bottom strip.  My best
guess is that it's just two images glued together, maybe taken at
different wavelengths.

-- 
To email me, substitute nowhere->runbox, invalid->com.

[toc] | [prev] | [next] | [standalone]

#100531

On Thu, Dec 17, 2015 at 4:30 AM, Peter Pearson
<pkpearson@nowhere.invalid> wrote:
> Agreed.  It's annoying when an agency goes to the trouble of making
> huge datasets available online, but fails to identify the format.
>
> But the 16-bits-per-pixel hypothesis is unlikely, given that each
> byte tends to echo its predecessor:
>
> 0000130 ffff ffff ffff ffff ffff ffff ffff ffff
> 0000140 ffff ffff ffff c0ff c1c0 c3c3 c3c3 c4c4
> 0000150 c4c4 c3c4 c3c3 c4c4 c3c3 c3c3 c3c3 c3c3
> 0000160 c4c4 c4c4 c5c4 c6c5 c7c7 c7c7 c5c5 c6c5

Hmm. With just a few exceptions. Maybe it's two channels or something
- is that what you mean by "taken at different wavelengths"?

Definitely it's begging for format identification from the source.

ChrisA

[toc] | [prev] | [next] | [standalone]

#100532

On Thu, 17 Dec 2015 04:37:26 +1100, Chris Angelico <rosuav@gmail.com> wrote:
> On Thu, Dec 17, 2015 at 4:30 AM, Peter Pearson
><pkpearson@nowhere.invalid> wrote:
>> Agreed.  It's annoying when an agency goes to the trouble of making
>> huge datasets available online, but fails to identify the format.
>>
>> But the 16-bits-per-pixel hypothesis is unlikely, given that each
>> byte tends to echo its predecessor:
>>
>> 0000130 ffff ffff ffff ffff ffff ffff ffff ffff
>> 0000140 ffff ffff ffff c0ff c1c0 c3c3 c3c3 c4c4
>> 0000150 c4c4 c3c4 c3c3 c4c4 c3c3 c3c3 c3c3 c3c3
>> 0000160 c4c4 c4c4 c5c4 c6c5 c7c7 c7c7 c5c5 c6c5
>
> Hmm. With just a few exceptions. Maybe it's two channels or something
> - is that what you mean by "taken at different wavelengths"?

Yes; but as described below, I now think they're taken at different times.

> Definitely it's begging for format identification from the source.

Agreed, again.  But it's hard to set this kind of problem aside.

I split it into two images, thusly:

>>> half = len(rawdata)/2
>>> Image.fromstring("L", (9896, 3298), rawdata[0:half]).save("temp3.png")
>>> Image.fromstring("L", (9896, 3298), rawdata[half:]).save("temp4.png")

Flipping between the resulting two images, one sees slight displacements
of the large-scale swirly structures, so I'm pretty sure the two images
correspond to slightly different times.  (I use the current GOES West
northern-hemisphere image as my desktop, so I'm pretty familiar with the
movements of atmospheric swirly thingies.)

This feels solved-enough to set aside now.

-- 
To email me, substitute nowhere->runbox, invalid->com.

[toc] | [prev] | [next] | [standalone]

#100534

Am 16.12.2015 um 18:30 schrieb Peter Pearson:
> On Thu, 17 Dec 2015 04:08:02 +1100, Chris Angelico <rosuav@gmail.com> wrote:
 >>
>> It could be 16 bits per pixel. Without knowing a lot more about the
>> source of the image and its format, it's hard to say with any
>> certainty.
>
> Agreed.  It's annoying when an agency goes to the trouble of making
> huge datasets available online, but fails to identify the format.

http://www.cpc.ncep.noaa.gov/products/global_precip/html/README

says (among other things):

| Each record is a 9896 x 3298 Fortran array of IR brightness
| temperatures that have been scaled to fit into 1-byte by subtracting
| "75" from each datum. Therefore it is necessary for the user to add a
| value of "75" to each data value when using the data.

HTH a little,

-- 
Robin Koch

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web