Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #95504 > unrolled thread

Storing dictionary locations as a string and using eval - alternatives?

Back to article view | Back to comp.lang.python

Contents

  Storing dictionary locations as a string and using eval - alternatives? Victor Hooi <victorhooi@gmail.com> - 2015-08-19 20:59 -0700
    Re: Storing dictionary locations as a string and using eval - alternatives? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-08-20 14:43 +1000
    Re: Storing dictionary locations as a string and using eval - alternatives? dieter <dieter@handshake.de> - 2015-08-20 07:57 +0200
    Re: Storing dictionary locations as a string and using eval - alternatives? Peter Otten <__peter__@web.de> - 2015-08-20 09:14 +0200

#95504 — Storing dictionary locations as a string and using eval - alternatives?

Hi,

I have a textfile with a bunch of JSON objects, one per line.

I'm looking at parsing each of these, and extract some metrics from each line.

I have a dict called "metrics_to_extract", containing the metrics I'm looking at extracting. In this, I store a name used to identify the metric, along with the location in the parsed JSON object.

Below is my code:

>>>>>>>>>>>>>>>>>>>>>>>
metrics_to_extract = {
    'current_connections': "server_status_json['connections']['current']",
    'resident_memory': "server_status_json['mem']['resident']"
}


def add_point(name, value, timestamp, tags):
    return {
        "measurement": name,
        "tags": tags,
        # "time": timestamp.isoformat(),
        "time": timestamp,
        "fields": {
            "value": float(value)
        }
    }

with open(input_file, 'r') as f:
    json_points = []
    for line in f:
        if line.startswith("{"):
            server_status_json = json.loads(line)
            # pp.pprint(server_status_json)
            # import ipdb; ipdb.set_trace()
            timestamp = server_status_json['localTime']
            tags = {
                'project': project,
                'hostname': server_status_json['host'],
                'version': server_status_json['version'],
                'storage_engine': server_status_json['storageEngine']['name']
            }

            for key, value in metrics_to_extract.items():
                json_points.append(add_point(key, eval(value), timestamp, tags))
            # client.write_points(json_points)
        else:
            print("non matching line")
>>>>>>>>>>>>>>>>>>>>>>>

My question is - I'm using "eval" in the above, with the nested location (e.g. "server_status_json['mem']['resident']") stored as a string.

I get the feeling this isn't particularly idiomatic or a great way of doing it - and would be keen to hear alternative suggestions?

Thanks,
Victor

[toc] | [next] | [standalone]

#95505

On Thursday 20 August 2015 13:59, Victor Hooi wrote:

> Hi,
> 
> I have a textfile with a bunch of JSON objects, one per line.
> 
> I'm looking at parsing each of these, and extract some metrics from each
> line.
> 
> I have a dict called "metrics_to_extract", containing the metrics I'm
> looking at extracting. In this, I store a name used to identify the
> metric, along with the location in the parsed JSON object.
> 
> Below is my code:
> 
>>>>>>>>>>>>>>>>>>>>>>>>
> metrics_to_extract = {
>     'current_connections': "server_status_json['connections']['current']",
>     'resident_memory': "server_status_json['mem']['resident']"
> }
> 
> 
> def add_point(name, value, timestamp, tags):
>     return {
>         "measurement": name,
>         "tags": tags,
>         # "time": timestamp.isoformat(),
>         "time": timestamp,
>         "fields": {
>             "value": float(value)
>         }
>     }

The function name is misleading and you should consider changing it. 
Misleading names make code harder to understand and maintain. The function 
doesn't "add point" at all, that implies that it should modify some data 
structure by adding (appending?) a point to it. Instead, it generates a dict 
containing (I presume) a data point.

But moving on...

Change metrics_to_extract above to this:

metrics_to_extract = {
    'current_connections': ['connections', 'current'],
    'resident_memory': ['mem', 'resident'],
    }

add this helper function:

def get_nested_items(obj, *names):
    """Return obj[name][name2] ... [nameN] for any list of names."""
    for name in names:
        obj = obj[name]
    return obj


and modify the code below as shown:


> with open(input_file, 'r') as f:
>     json_points = []
>     for line in f:
>         if line.startswith("{"):
[...]
>             for key, value in metrics_to_extract.items():
>                 json_points.append(add_point(key, eval(value), timestamp,
>                 tags))

            for key, names in metrics_to_extract.items():
                value = get_nested_items(server_status_json, *names)
                point = add_point(key, value, timestamp, tags)
                json_points.append(point)




-- 
Steven

[toc] | [prev] | [next] | [standalone]

#95506

Victor Hooi <victorhooi@gmail.com> writes:
> ...
I did not read your message body - just your "subject".

Instead of (the potentially insecure) "eval", I use "json.loads"
(combined with a "json.dumps" to get the "json" representation).
See the "json" module documentation.

[toc] | [prev] | [next] | [standalone]

#95507

dieter wrote:

> Victor Hooi <victorhooi@gmail.com> writes:
>> ...
> I did not read your message body - just your "subject".

That was a mistake ;)

> Instead of (the potentially insecure) "eval", I use "json.loads"
> (combined with a "json.dumps" to get the "json" representation).
> See the "json" module documentation.

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web