Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #95717 > unrolled thread

Casting to a "number" (both int and float)?

Back to article view | Back to comp.lang.python

Contents

  Casting to a "number" (both int and float)? Victor Hooi <victorhooi@gmail.com> - 2015-08-27 21:57 -0700
    Re: Casting to a "number" (both int and float)? Victor Hooi <victorhooi@gmail.com> - 2015-08-27 22:04 -0700
      Re: Casting to a "number" (both int and float)? "Sven R. Kunze" <srkunze@mail.de> - 2015-08-28 18:09 +0200
        Re: Casting to a "number" (both int and float)? Robin Koch <robin.koch@t-online.de> - 2015-08-28 18:15 +0200
          Re: Casting to a "number" (both int and float)? "Sven R. Kunze" <srkunze@mail.de> - 2015-08-30 13:38 +0200
          Re: Casting to a "number" (both int and float)? Steven D'Aprano <steve@pearwood.info> - 2015-08-30 22:00 +1000
          Re: Casting to a "number" (both int and float)? Mark Lawrence <breamoreboy@yahoo.co.uk> - 2015-08-30 13:55 +0100
    Re: Casting to a "number" (both int and float)? Ben Finney <ben+python@benfinney.id.au> - 2015-08-28 15:44 +1000
      Re: Casting to a "number" (both int and float)? Jussi Piitulainen <harvestingn@makes.address.invalid> - 2015-08-28 09:03 +0300
      Re: Casting to a "number" (both int and float)? Jussi Piitulainen <harvesting@makes.address.invalid> - 2015-08-28 09:15 +0300
        Re: Casting to a "number" (both int and float)? Victor Hooi <victorhooi@gmail.com> - 2015-08-28 00:56 -0700
          Re: Casting to a "number" (both int and float)? Chris Angelico <rosuav@gmail.com> - 2015-08-28 19:23 +1000
    Re: Casting to a "number" (both int and float)? Jussi Piitulainen <harvesting@makes.address.invalid> - 2015-08-28 08:46 +0300
    Re: Casting to a "number" (both int and float)? random832@fastmail.us - 2015-08-28 01:54 -0400
    Re: Casting to a "number" (both int and float)? Laura Creighton <lac@openend.se> - 2015-08-28 07:59 +0200

#95717 — Casting to a "number" (both int and float)?

I'm reading JSON output from an input file, and extracting values.

Many of the fields are meant to be numerical, however, some fields are wrapped in a "floatApprox" dict, which messed with my parsing.

For example:

{
    "hostname": "example.com",
    "version": "3.0.5",
    "pid": {
        "floatApprox": 18403
    }
    "network": {
        "bytesIn": 123123,
        "bytesOut": {
            "floatApprox": 213123123
        }
}

The floatApprox wrapping appears to happen sporadically in the input.

I'd like to find a way to deal with this robustly.

For example, I have the following function:

def strip_floatApprox_wrapping(field):
    # Extracts a integer value from a field. Workaround for the float_approx wrapping.
    try:
        return int(field)
    except TypeError:
        return int(field['floatApprox'])

which I can then call on each field I want to extract.

However, this relies on casting to int, which will only work for ints - for some fields, they may actually be floats, and I'd like to preserve that if possible.

(I know there's a isnumber() field - but you can only call that on a string - so if I do hit a floatApprox field, it will trigger a AttributeError exception, which seems a bit clunky to handle).

def strip_floatApprox_wrapping(field):
    # Extracts a integer value from a field. Workaround for the float_approx wrapping.
    try:
        if field.isnumeric():
            return field
    except AttributeError:
        return field['floatApprox']

Is there a way to re-write strip_floatApprox_wrapping to handle both ints/floats, and preserve the original format?

Or is there a more elegant way to deal with the arbitrary nesting with floatApprox?

[toc] | [next] | [standalone]

#95718

Actually, I've just realised, if I just test for numeric or try to cast to ints, this will break for string fields.

As in, the intention is to call strip_floatAprox_wrapping on all the fields I'm parsing, and have it deal with the floatApprox dict wrapping, whether the contents are numbers or strings (although strings would not be wrapped in floatApprox).

On Friday, 28 August 2015 14:58:01 UTC+10, Victor Hooi  wrote:
> I'm reading JSON output from an input file, and extracting values.
> 
> Many of the fields are meant to be numerical, however, some fields are wrapped in a "floatApprox" dict, which messed with my parsing.
> 
> For example:
> 
> {
>     "hostname": "example.com",
>     "version": "3.0.5",
>     "pid": {
>         "floatApprox": 18403
>     }
>     "network": {
>         "bytesIn": 123123,
>         "bytesOut": {
>             "floatApprox": 213123123
>         }
> }
> 
> The floatApprox wrapping appears to happen sporadically in the input.
> 
> I'd like to find a way to deal with this robustly.
> 
> For example, I have the following function:
> 
> def strip_floatApprox_wrapping(field):
>     # Extracts a integer value from a field. Workaround for the float_approx wrapping.
>     try:
>         return int(field)
>     except TypeError:
>         return int(field['floatApprox'])
> 
> which I can then call on each field I want to extract.
> 
> However, this relies on casting to int, which will only work for ints - for some fields, they may actually be floats, and I'd like to preserve that if possible.
> 
> (I know there's a isnumber() field - but you can only call that on a string - so if I do hit a floatApprox field, it will trigger a AttributeError exception, which seems a bit clunky to handle).
> 
> def strip_floatApprox_wrapping(field):
>     # Extracts a integer value from a field. Workaround for the float_approx wrapping.
>     try:
>         if field.isnumeric():
>             return field
>     except AttributeError:
>         return field['floatApprox']
> 
> Is there a way to re-write strip_floatApprox_wrapping to handle both ints/floats, and preserve the original format?
> 
> Or is there a more elegant way to deal with the arbitrary nesting with floatApprox?

[toc] | [prev] | [next] | [standalone]

#95737

Hey Victor,

for proper parsing into native Python types, I would recommend YAML.

Also also supports (besides int vs. float) dates and datetimes.

Cheers,
Sven

On 28.08.2015 07:04, Victor Hooi wrote:
> Actually, I've just realised, if I just test for numeric or try to cast to ints, this will break for string fields.
>
> As in, the intention is to call strip_floatAprox_wrapping on all the fields I'm parsing, and have it deal with the floatApprox dict wrapping, whether the contents are numbers or strings (although strings would not be wrapped in floatApprox).
>
> On Friday, 28 August 2015 14:58:01 UTC+10, Victor Hooi  wrote:
>> I'm reading JSON output from an input file, and extracting values.
>>
>> Many of the fields are meant to be numerical, however, some fields are wrapped in a "floatApprox" dict, which messed with my parsing.
>>
>> For example:
>>
>> {
>>      "hostname": "example.com",
>>      "version": "3.0.5",
>>      "pid": {
>>          "floatApprox": 18403
>>      }
>>      "network": {
>>          "bytesIn": 123123,
>>          "bytesOut": {
>>              "floatApprox": 213123123
>>          }
>> }
>>
>> The floatApprox wrapping appears to happen sporadically in the input.
>>
>> I'd like to find a way to deal with this robustly.
>>
>> For example, I have the following function:
>>
>> def strip_floatApprox_wrapping(field):
>>      # Extracts a integer value from a field. Workaround for the float_approx wrapping.
>>      try:
>>          return int(field)
>>      except TypeError:
>>          return int(field['floatApprox'])
>>
>> which I can then call on each field I want to extract.
>>
>> However, this relies on casting to int, which will only work for ints - for some fields, they may actually be floats, and I'd like to preserve that if possible.
>>
>> (I know there's a isnumber() field - but you can only call that on a string - so if I do hit a floatApprox field, it will trigger a AttributeError exception, which seems a bit clunky to handle).
>>
>> def strip_floatApprox_wrapping(field):
>>      # Extracts a integer value from a field. Workaround for the float_approx wrapping.
>>      try:
>>          if field.isnumeric():
>>              return field
>>      except AttributeError:
>>          return field['floatApprox']
>>
>> Is there a way to re-write strip_floatApprox_wrapping to handle both ints/floats, and preserve the original format?
>>
>> Or is there a more elegant way to deal with the arbitrary nesting with floatApprox?

[toc] | [prev] | [next] | [standalone]

#95738

Am 28.08.2015 um 18:09 schrieb Sven R. Kunze:

 >> I'm reading JSON output from an input file, and extracting values.
 >
> for proper parsing into native Python types, I would recommend YAML.

"What's the best way to get from A to B?"
"I recommend starting at C."

- Every other usenet-discussion.

-- 
Robin Koch

[toc] | [prev] | [next] | [standalone]

#95764

Well, it comes down to "can he switch from JSON to YAML" in the first 
place as he is going to parse "by-third-party" generated json. So, if he 
can persuade MongoDB to output YAML instead, he would be all fine.

Just my 2 cents.

On 28.08.2015 18:15, Robin Koch wrote:
> Am 28.08.2015 um 18:09 schrieb Sven R. Kunze:
>
> >> I'm reading JSON output from an input file, and extracting values.
> >
>> for proper parsing into native Python types, I would recommend YAML.
>
> "What's the best way to get from A to B?"
> "I recommend starting at C."
>
> - Every other usenet-discussion.
>

[toc] | [prev] | [next] | [standalone]

#95765

On Sat, 29 Aug 2015 02:15 am, Robin Koch wrote:

> Am 28.08.2015 um 18:09 schrieb Sven R. Kunze:
> 
>  >> I'm reading JSON output from an input file, and extracting values.
>  >
>> for proper parsing into native Python types, I would recommend YAML.
> 
> "What's the best way to get from A to B?"
> "I recommend starting at C."
> 
> - Every other usenet-discussion.


"I have a pig's ear. What's the easiest way to turn it into a silk purse?"

"I recommend starting with with something else."

For the non-native English speakers here, we have a proverb "You cannot turn
a pig's ear into a silk purse", a metaphor for the impossibility of making
something (or someone) of good quality from a inferior stock.

As programmers, we must recognise the importance of picking the right data
structures. For instance, if we want fast key lookups, we would pick a
dict, not a large string that needs to be slowly parsed each and every time
we perform a lookup. So the question:

"I have a large string of the form 'key:value', one key per line, how do I
quickly look up a key and get the value?"

would rightly get the answer "You don't. Use a dict." Possibly that means a
once-off conversion from string to dict, or possibly it means convincing
the source of the data to return a dict instead.

In this case, I have no opinion on whether JSON or YAML (or indeed something
else) is a better solution. But it is completely legitimate to suggest that
the current solution is sub-optimal and another would be better.




-- 
Steven

[toc] | [prev] | [next] | [standalone]

#95766

".esnes sekam ti wonk uoY  .daer ot ysae os sgniht ekam seod yllaer ti 
,niaga gnitsop potrof s'knahT"

On 30/08/2015 12:38, Sven R. Kunze wrote:
> Well, it comes down to "can he switch from JSON to YAML" in the first
> place as he is going to parse "by-third-party" generated json. So, if he
> can persuade MongoDB to output YAML instead, he would be all fine.
>
> Just my 2 cents.
>
> On 28.08.2015 18:15, Robin Koch wrote:
>> Am 28.08.2015 um 18:09 schrieb Sven R. Kunze:
>>
>> >> I'm reading JSON output from an input file, and extracting values.
>> >
>>> for proper parsing into native Python types, I would recommend YAML.
>>
>> "What's the best way to get from A to B?"
>> "I recommend starting at C."
>>
>> - Every other usenet-discussion.
>>
>

-- 
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence

[toc] | [prev] | [next] | [standalone]

#95719

Victor Hooi <victorhooi@gmail.com> writes:

> Many of the fields are meant to be numerical, however, some fields are
> wrapped in a "floatApprox" dict, which messed with my parsing.

The examples you give of ‘floatApprox’ are not dicts, so I'm not sure
quite what that means.

> For example:
>
> {
>     "hostname": "example.com",
>     "version": "3.0.5",
>     "pid": {
>         "floatApprox": 18403
>     }
>     "network": {
>         "bytesIn": 123123,
>         "bytesOut": {
>             "floatApprox": 213123123
>         }
> }
>
> The floatApprox wrapping appears to happen sporadically in the input.

What do you mean by “wrapping”?

Can you show exactly what output values you would expect for these
example inputs?

> However, this relies on casting to int, which will only work for ints
> - for some fields, they may actually be floats, and I'd like to
> preserve that if possible.

(A terminology question: Note that “cast to int” isn't something that
happens in Python. You never “cast” an object as a different type, the
object is exactly what type it is and no other.

The example code you showed is not casting, but creating. Calling the
‘int’ type makes a new object of that type. Python doesn't have “cast”
as a concept.)

The question that needs to be answered is, how do you know what values
are “actually” floats?

In JSON there is no distinction at all, the only numeric type is
‘float’. What information is there in the input that can be used to know
which values should result in an ‘int’ instance, versus values that
should result in a ‘float’ instance?

-- 
 \                         “I'm a great lover, I'll bet.” —Emo Philips |
  `\                                                                   |
_o__)                                                                  |
Ben Finney

[toc] | [prev] | [next] | [standalone]

#95724

Ben Finney writes:

> Victor Hooi writes:
>
>> Many of the fields are meant to be numerical, however, some fields are
>> wrapped in a "floatApprox" dict, which messed with my parsing.
>
> The examples you give of ‘floatApprox’ are not dicts, so I'm not sure
> quite what that means.

I took the distinction to be between 123123 and {"floatApprox":
213123123}. The former looks like an int, the latter like a dict.

>>     "network": {
>>         "bytesIn": 123123,
>>         "bytesOut": {
>>             "floatApprox": 213123123
>>         }

> What do you mean by “wrapping”?

I read it so that the bytesOut value, 213123123, is "wrapped" in
{"floatApprox": }, and Victor wants just the number inside.

[- -]

[toc] | [prev] | [next] | [standalone]

#95725

Ben Finney writes:

> Victor Hooi writes:
[- -]
>> For example:
>>
>> {
>>     "hostname": "example.com",
>>     "version": "3.0.5",
>>     "pid": {
>>         "floatApprox": 18403
>>     }
>>     "network": {
>>         "bytesIn": 123123,
>>         "bytesOut": {
>>             "floatApprox": 213123123
>>         }
>> }

[- -]

> In JSON there is no distinction at all, the only numeric type is
> ‘float’. What information is there in the input that can be used to
> know which values should result in an ‘int’ instance, versus values
> that should result in a ‘float’ instance?

I seem to get ints in the example data.

    >>> json.load(io.StringIO('{"floatApprox":31213}'))
    {'floatApprox': 31213}

[toc] | [prev] | [next] | [standalone]

#95730

Hi,

Thanks heaps to everybody for their suggestions/advice =).

Currently I'm using this:

def strip_floatApprox_wrapping(field):
    # Extracts a integer value from a field. Workaround for the float_approx wrapping.
    if isinstance(field, dict):
        return field['floatApprox']
    else:
        return field

I was a little hesitant to go down that path (using isinstance()) since it seems a bit "un-Pythonic" but it seems to do what I want in a minimal amount of code.

Somebody suggested going through and transforming the whole output from json.loads() in a single pass - I don't actually need *all* the fields, just a fair number of them - is there a particularly strong reason stripping out the floatApprox()'s first is a good approach?

Also, to answer somebody's question - yes, this is MongoDB, specifically the output from db.serverStatus(). The logging and reporting consistency in MongoDB is...quirky, shall we say.

Cheers,
Victor

On Friday, 28 August 2015 16:15:21 UTC+10, Jussi Piitulainen  wrote:
> Ben Finney writes:
> 
> > Victor Hooi writes:
> [- -]
> >> For example:
> >>
> >> {
> >>     "hostname": "example.com",
> >>     "version": "3.0.5",
> >>     "pid": {
> >>         "floatApprox": 18403
> >>     }
> >>     "network": {
> >>         "bytesIn": 123123,
> >>         "bytesOut": {
> >>             "floatApprox": 213123123
> >>         }
> >> }
> 
> [- -]
> 
> > In JSON there is no distinction at all, the only numeric type is
> > 'float'. What information is there in the input that can be used to
> > know which values should result in an 'int' instance, versus values
> > that should result in a 'float' instance?
> 
> I seem to get ints in the example data.
> 
>     >>> json.load(io.StringIO('{"floatApprox":31213}'))
>     {'floatApprox': 31213}

[toc] | [prev] | [next] | [standalone]

#95733

On Fri, Aug 28, 2015 at 5:56 PM, Victor Hooi <victorhooi@gmail.com> wrote:
>
> Currently I'm using this:
>
> def strip_floatApprox_wrapping(field):
>     # Extracts a integer value from a field. Workaround for the float_approx wrapping.
>     if isinstance(field, dict):
>         return field['floatApprox']
>     else:
>         return field
>
> I was a little hesitant to go down that path (using isinstance()) since it seems a bit "un-Pythonic" but it seems to do what I want in a minimal amount of code.

It'd not be as Pythonic if this were an API, but since you just
converted this from JSON, you can be fairly confident of what the data
types are. You're dealing with your program's input, and that input
can come in a specific number of forms; so you cope with each form.
This is a perfectly acceptable and reasonable way to write code.

ChrisA

[toc] | [prev] | [next] | [standalone]

#95720

Victor Hooi writes:

> I'm reading JSON output from an input file, and extracting values.
>
> Many of the fields are meant to be numerical, however, some fields are
> wrapped in a "floatApprox" dict, which messed with my parsing.
>
> For example:
>
> {
>     "hostname": "example.com",
>     "version": "3.0.5",
>     "pid": {
>         "floatApprox": 18403
>     }
>     "network": {
>         "bytesIn": 123123,
>         "bytesOut": {
>             "floatApprox": 213123123
>         }
> }
>
> The floatApprox wrapping appears to happen sporadically in the input.
>
> I'd like to find a way to deal with this robustly.

So you want a value as is when it is a number - either an int or a
float. And otherwise the value is a dict and you want the value at a
known key. Here's a literal-minded expression of the spec:

   ( o if isinstance(o, (int, float)) else o['floatApprox'] )

> For example, I have the following function:
>
> def strip_floatApprox_wrapping(field):
>     # Extracts a integer value from a field. Workaround for the
>     # float_approx wrapping.
>     try:
>         return int(field)
>     except TypeError:
>         return int(field['floatApprox'])
>
> which I can then call on each field I want to extract.
>
> However, this relies on casting to int, which will only work for ints
> - for some fields, they may actually be floats, and I'd like to
> preserve that if possible.

You could try int first, float second; you should still return the
actual value.

   try:
     int(field)
     return field
   except TypeError: pass
   try:
     float(field)
     return field:
   except TypeError:
     return field['floatApprox']

There are a couple of reasons to return the actual value instead of the
result of the "cast". First, it appears to be what you want :) Second,
it allows you to have some confidence that the data actually is numeric
and not, say, strings that happen to look like numbers.

> (I know there's a isnumber() field - but you can only call that on a
> string - so if I do hit a floatApprox field, it will trigger a
> AttributeError exception, which seems a bit clunky to handle).
>
> def strip_floatApprox_wrapping(field):
>     # Extracts a integer value from a field. Workaround for the
>     # float_approx wrapping.
>     try:
>         if field.isnumeric():
>             return field
>     except AttributeError:
>         return field['floatApprox']

I take it you haven't tried this and the fields in question actually are
numeric, not strings. That exception would be taken for a wrong reason.

You could do the following, but str.isnumeric does not do what you want.
For example, "3.14".isnumeric() is as False as "{}".isnumeric().

    if str(field).isnumeric():
        return field
    else:
        return field['floatApprox']

> Is there a way to re-write strip_floatApprox_wrapping to handle both
> ints/floats, and preserve the original format?
>
> Or is there a more elegant way to deal with the arbitrary nesting with
> floatApprox?

Consider ... if isinstance(field, (int, float)) else ... .

[toc] | [prev] | [next] | [standalone]

#95721

On Fri, Aug 28, 2015, at 00:57, Victor Hooi wrote:
> I'm reading JSON output from an input file, and extracting values.
>
> Many of the fields are meant to be numerical, however, some fields are
> wrapped in a "floatApprox" dict, which messed with my parsing.

> Is there a way to re-write strip_floatApprox_wrapping to handle both
> ints/floats, and preserve the original format?

You could do simply isinstance(field, (int, float)).

> Or is there a more elegant way to deal with the arbitrary nesting with
> floatApprox?

I'd probably just recursively walk through the dictionary finding these
and removing them.

def clean_floatapprox(obj):
  if type(obj) is dict:
    for key, value in obj.items():
      if type(value) is dict:
        if len(value) == 1 and 'floatApprox' in value:
          obj[key] = value['floatApprox']
        else
          clean_floatapprox(obj)

Since it's a JSON result I didn't bother with making a copy or dealing
with the possibility of circular references for this example function,
so you'll need to add to this if you have to deal with those things.

Incidentally, do you ever get one that represents a very large value and
has more fields than just floatApprox? I googled this and it looks like
it's a MongoDB thing.

The example given is {"floatApprox" : 9223372036854776000,
    "top" : 2147483647,  "bottom" : 4294967295}

Actually representing the value 9223372036854775807. If top and bottom
are found the result should be top << 32 | bottom. I don't know how
large negative numbers are represented.

[toc] | [prev] | [next] | [standalone]

#95723

I suspect your code will have these 2 lines in it somewhere ...

if isinstance(field, dict):
   return int(field['floatApprox'])

using isinstance() or type() is generally frowned upon because
it breaks duck typing, and makes it necessary for you to write more
code every time somebody wants to feed your code slightly different
arguments.  Your code breaks for no particularly good reason.

But that's not your problem here.  You are parsing things.  And right
now it is your job to know exactly what type things are -- that _is_
the job.  Should somebody start feeding you field['Money'] you don't
want things to quietly assume that pretending it is a float will work,
when you really want a Decimal -- raising a TypeError about 'I don't
recognise this type' is probably what you want to do here.

Laura

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web