Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #102546 > unrolled thread

realtime output and csv files

Back to article view | Back to comp.lang.python

Contents

  realtime output and csv files lucan <lucan@NNN.it> - 2016-02-05 20:49 +0100
    Re: realtime output and csv files Bernardo Sulzbach <mafagafogigante@gmail.com> - 2016-02-05 17:57 -0200
      Re: realtime output and csv files lucan <lucan@NNN.it> - 2016-02-05 22:09 +0100
        Re: realtime output and csv files Bernardo Sulzbach <mafagafogigante@gmail.com> - 2016-02-05 19:13 -0200
        Re: realtime output and csv files Joel Goldstick <joel.goldstick@gmail.com> - 2016-02-05 16:43 -0500
        Re: realtime output and csv files Bernardo Sulzbach <mafagafogigante@gmail.com> - 2016-02-05 19:51 -0200
          Re: realtime output and csv files lucan <lucan@NNN.it> - 2016-02-06 14:40 +0100
        Re: realtime output and csv files Joel Goldstick <joel.goldstick@gmail.com> - 2016-02-05 17:12 -0500
    Re: realtime output and csv files Tim Chase <python.list@tim.thechases.com> - 2016-02-05 22:27 -0600
    Re: realtime output and csv files Bernardo Sulzbach <mafagafogigante@gmail.com> - 2016-02-06 02:53 -0200
    Re: realtime output and csv files Tim Chase <python.list@tim.thechases.com> - 2016-02-06 06:45 -0600

#102546 — realtime output and csv files

I'm new of python adn I'm using it only to complete some experiments.
I'm reading a series of data from various sensors and in the meantime 
I'm writing datas on a file. I would like to print output in realtime 
(or refresh it when I need) but the problem is that I'm writing on a 
file every x seconds (my timestep).
Anyway from the moment that datas are scientific value is it correct to 
write on a file using str(temp) and separating with ","?
I need a csv file to read it with some data analysis softwares.


...
now = datetime.datetime.now()
dayexperiment = str(now.day)+str(now.month)+str(now.year)

myfilename ="myData"+dayexperiment+".dat"

index = 0
count = 1

timestep = 10


with open(myfilename,'w') as f:
f.write('#Index Time Value')
while True:
	reading = ...
     	...
	temp=...

         print 'Temp{0:0.3F}*C'.format(temp)

	now = datetime.datetime.now()
	file.write('\n'+str(index))
	
f.write(str(index)+','+str(now.hour)+':'+str(now.minute)+', '+str(temp))
		
		index +=1
		time.sleep(timestep)

[toc] | [next] | [standalone]

#102547

On 02/05/2016 05:49 PM, lucan wrote:
> Anyway from the moment that datas are scientific value is it correct to
> write on a file using str(temp) and separating with ","?
> I need a csv file to read it with some data analysis softwares.
>

What do you mean? What is "datas"? What do you mean by "correct"?

CSVs is essentially text separated by commas, so you likely do not need 
any library to write it "Just separating with ','" should work if you 
are formatting them correctly.

[toc] | [prev] | [next] | [standalone]

#102558

> What do you mean? What is "datas"? What do you mean by "correct"?

"datas" I mean the values for example temperature = 20.4 (so they are 
floating point)

Index time temp
1 10:24 20.4
2 10:25 20.6
...

I wonder if this is correct "my way" to write a csv file:

file.write('\n'+str(index)) 
f.write(str(index)+','+str(now.hour)+':'+str(now.minute)+','+str(temp))

Or if there is a better way to do that.

> CSVs is essentially text separated by commas, so you likely do not need
> any library to write it "Just separating with ','" should work if you
> are formatting them correctly.

ok.

[toc] | [prev] | [next] | [standalone]

#102559

On 02/05/2016 07:09 PM, lucan wrote:
>
>> What do you mean? What is "datas"? What do you mean by "correct"?
>
> "datas" I mean the values for example temperature = 20.4 (so they are
> floating point)
>
> Index time temp
> 1 10:24 20.4
> 2 10:25 20.6
> ...
>
> I wonder if this is correct "my way" to write a csv file:
>
> file.write('\n'+str(index))
> f.write(str(index)+','+str(now.hour)+':'+str(now.minute)+','+str(temp))
>
> Or if there is a better way to do that.
>

If you got your newlines and headers (if the other program requires it) 
right, this should be all you need.

[toc] | [prev] | [next] | [standalone]

#102562

On Fri, Feb 5, 2016 at 4:13 PM, Bernardo Sulzbach <mafagafogigante@gmail.com
> wrote:

> On 02/05/2016 07:09 PM, lucan wrote:
>
>>
>> What do you mean? What is "datas"? What do you mean by "correct"?
>>>
>>
>> "datas" I mean the values for example temperature = 20.4 (so they are
>> floating point)
>>
>> Index time temp
>> 1 10:24 20.4
>> 2 10:25 20.6
>> ...
>>
>> I wonder if this is correct "my way" to write a csv file:
>>
>> file.write('\n'+str(index))
>> f.write(str(index)+','+str(now.hour)+':'+str(now.minute)+','+str(temp))
>>
>
You might want to do this:

f.write("%d, %2d:%2d, %.1f" % (index, now.hour, now.minute, temp))

In my test:

>>> print("%d, %2d:%2d, %.1f" % (1,10,24,20.4))
1, 10:24, 20.4

This uses the original python formatting method.  There is a newer one as
well

>
>> Or if there is a better way to do that.
>>
>>
> If you got your newlines and headers (if the other program requires it)
> right, this should be all you need.
> --
> https://mail.python.org/mailman/listinfo/python-list
>



-- 
Joel Goldstick
http://joelgoldstick.com/stats/birthdays

[toc] | [prev] | [next] | [standalone]

#102563

On 02/05/2016 07:43 PM, Joel Goldstick wrote:
>>>> print("%d, %2d:%2d, %.1f" % (1,10,24,20.4))
> 1, 10:24, 20.4

Let us be more careful there. Although CSV has no formal specification 
(according to the IETF), *those spaces are not good*.

It is **very unlikely** that they will cause issues, but 1,10:24,20.4 is 
*safer* and - to my eyes - better CSV.

I wouldn't be surprised if a parser treated a value as text only because 
it has spaces on it.

For OP, if you are going for this, I - personally - suggest sticking to 
"%d,%2d:%2d,%.1f".

[toc] | [prev] | [next] | [standalone]

#102581

> I wouldn't be surprised if a parser treated a value as text only because
> it has spaces on it.
>
> For OP, if you are going for this, I - personally - suggest sticking to
> "%d,%2d:%2d,%.1f".

you're rightin fact importing datas in spreadsheet I've had some 
problems. I'll follow this suggestion. tnks a lot! :)

[toc] | [prev] | [next] | [standalone]

#102566

On Fri, Feb 5, 2016 at 4:51 PM, Bernardo Sulzbach <mafagafogigante@gmail.com
> wrote:

> On 02/05/2016 07:43 PM, Joel Goldstick wrote:
>
>> print("%d, %2d:%2d, %.1f" % (1,10,24,20.4))
>>>>>
>>>> 1, 10:24, 20.4
>>
>
> Let us be more careful there. Although CSV has no formal specification
> (according to the IETF), *those spaces are not good*.
>
> It is **very unlikely** that they will cause issues, but 1,10:24,20.4 is
> *safer* and - to my eyes - better CSV.
>
> I wouldn't be surprised if a parser treated a value as text only because
> it has spaces on it.
>
> For OP, if you are going for this, I - personally - suggest sticking to
> "%d,%2d:%2d,%.1f".


Good point!

>
> --
> https://mail.python.org/mailman/listinfo/python-list
>



-- 
Joel Goldstick
http://joelgoldstick.com/stats/birthdays

[toc] | [prev] | [next] | [standalone]

#102571

On 2016-02-05 17:57, Bernardo Sulzbach wrote:
> CSVs is essentially text separated by commas, so you likely do not
> need any library to write it "Just separating with ','" should work
> if you are formatting them correctly.

> https://mail.python.org/mailman/listinfo/python-list
And even if you have things to escape or format correctly, the stdlib
has a "csv" module that makes this trivially easy:

  data = [
    (1, 3.14, "hello"),
    (2, 1.41, "goodbye"),
    ]
  from csv import writer
  with open("out.csv", "wb") as f:
    w = writer(f)
    w.writerows(data)
    # for row in data:
    #     w.writerow(data)

-tkc

[toc] | [prev] | [next] | [standalone]

#102572

On Sat, Feb 6, 2016 at 2:27 AM, Tim Chase <python.list@tim.thechases.com> wrote:
> On 2016-02-05 17:57, Bernardo Sulzbach wrote:
>> CSVs is essentially text separated by commas, so you likely do not
>> need any library to write it "Just separating with ','" should work
>> if you are formatting them correctly.
>
>> https://mail.python.org/mailman/listinfo/python-list
> And even if you have things to escape or format correctly, the stdlib
> has a "csv" module that makes this trivially easy:
>

I supposed it had one. Obviously, I've never used it myself, otherwise
I would be sure about its existence. Nice to know about it, although I
assume that for many data transfers it will not be painless if you
need several escapes, when CSV starts to become complicated it starts
to fail because of quirky and naive parsers.

[toc] | [prev] | [next] | [standalone]

#102582

On 2016-02-06 02:53, Bernardo Sulzbach wrote:
>> And even if you have things to escape or format correctly, the
>> stdlib has a "csv" module that makes this trivially easy:
>>  
> 
> I supposed it had one. Obviously, I've never used it myself,
> otherwise I would be sure about its existence. Nice to know about
> it, although I assume that for many data transfers it will not be
> painless if you need several escapes, when CSV starts to become
> complicated it starts to fail because of quirky and naive parsers.

If you read up on the csv module's help, it has all manner of knobs
to twiddle when it comes to the "dialect" of CSV:

- delimiters:  comma is the default, but you can specify others like
  tab or pipe)

- quote character:  double-quote is the default, but you
  can use single-quote or otherwise

- escaping: doubling up the quote-character is the default but I think
  you can specify other schemes


There are others, but that's what I have to tweak most often.

Finally, there's a sniffer in the module, so even if you don't know
the format, you can make an intelligent guess about it:

  with open('example.csv', 'rb') as csvfile:
    dialect = csv.Sniffer().sniff(csvfile.read(1024*4))
    csvfile.seek(0)
    reader = csv.reader(csvfile, dialect)

The only edge case I've hit regularly is that it doesn't (didn't? not
sure if it's been fixed) do the right thing on CSV files containing a
single column, as it would try to accept a common alphanumeric
character as the delimiter instead of making the better assumption
that it was a single column.

-tkc

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web