Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]


Groups > comp.soft-sys.math.mathematica > #5830 > unrolled thread

Re: runs slowly?

Started byMichael Stern <nycstern@gmail.com>
First post2011-10-24 09:13 +0000
Last post2011-10-27 10:30 +0000
Articles 4 — 3 participants

Back to article view | Back to comp.soft-sys.math.mathematica

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.


Contents

  Re: runs slowly? Michael Stern <nycstern@gmail.com> - 2011-10-24 09:13 +0000
    Re: runs slowly? Armand Tamzarian <mike.honeychurch@gmail.com> - 2011-10-25 10:22 +0000
      Re: runs slowly? Armand Tamzarian <mike.honeychurch@gmail.com> - 2011-10-26 21:45 +0000
    Re: runs slowly? DrMajorBob <btreat1@austin.rr.com> - 2011-10-27 10:30 +0000

#5830 — Re: runs slowly?

FromMichael Stern <nycstern@gmail.com>
Date2011-10-24 09:13 +0000
SubjectRe: runs slowly?
Message-ID<j83a7d$k07$1@smc.vnet.net>
All the date functions in Mathematica are extremely slow. In benchmarks I sent to Wolfram a few months ago (and some of which can be seen at
http://www.wheels.org/monkeywrench/?p=453), the date functions in Mathematica ran about 1/82 the speed of the same functions in Visual Basic for Applications. If you know the format of the dates you are converting, you may be able to roll your own date functions much faster than Wolfram's. I have encouraged them to improve this code, but they will probably pay closer attention if others complain as well.

Michael


Sent from my iPad

On Oct 23, 2011, at 6:25 AM, Robert McHugh <bob_mchugh_2000@yahoo.com> wrote:

> The following code takes over a minute to run on my machine.  Is this expected behavior?
>
> t = {"14-May-10 09:58:05", {"Day", "-", "MonthNameShort", "-", "YearShort", " ", "Hour", ":", "Minute", ":", "Second"}};
> tList = Table[t, {i, 100000}];
> a = Timing[AbsoluteTime[#] & /@ tList ;]
>
> For reference, an operation like the following takes less than a tenth of a second. (Of course this second example needs quite a bit of modification to provide a correct answer, but it does show how fast the program can operate on a large list.)
> t = {14, 5, 10, 9, 58, 05};
> tList = Table[t, {i, 100000}];
> a = Timing[( ((#[[3]] 0 + #[[2]] 30 + #[[1]]24) + #[[4]]) 60 + #[[5]]) 60 + #[[6]] & /@ tList;]
>
> Some background: am analyzing some historical data (about 500 000 records, one data point a minute for about a year) and am making a few utilities to retrieve the data for any given time interval.   My original plan was to change the time stamp to absolute time and then use a select statement.  This step in the above example, changing the time stamps to absolute time, is the rate limiting step in the code (everything else runs in about 5 seconds).
>
> Was wondering if someone could explain why AbsoluteTime[] is relatively slow operation and perhaps some faster operations for date and time comparisons.
>
> Thanks
>

[toc] | [next] | [standalone]


#5853

FromArmand Tamzarian <mike.honeychurch@gmail.com>
Date2011-10-25 10:22 +0000
Message-ID<j862lo$5t7$1@smc.vnet.net>
In reply to#5830
On Oct 24, 8:13 pm, Michael Stern <nycst...@gmail.com> wrote:
> All the date functions in Mathematica are extremely slow. In benchmarks I sent to Wolfram a few months ago (and some of which can be seen athttp://www.wheels.org/monkeywrench/?p=453), the date functions in Mathematica ran about 1/82 the speed of the same functions in Visual Basic for Application s. If you know the format of the dates you are converting, you may be able to roll your own date functions much faster than Wolfram's. I have encourag ed them to improve this code, but they will probably pay closer attention if others complain as well.
>
> Michael
>
> Sent from my iPad


I'd have to agree with you.

If you're doing thousands or tens of thousands of calculations daily
that require date manipulations these are often the rate determining
step. It seems like an example where a "super function" is a
significant drawback. What I mean is that if you are working with a
constant format you can, as Michael says above, write your own
function that is between one and two orders of magnitude faster than
the built in function. For the volumes of calculations that you might
find yourself doing if working in a financial entity, these efficiency
gains can be the difference between Mathematica being a competitive software
option or not.

A competitor software might have a different function for each date
format, analogous to the roll your own function that you write,
whereas Mathematica might have one super function that handles all formats
etc. etc. (but slowly). For prototyping this is not likely to be a
problem. For large scale high volumes of calculations the competitor
wins on efficiency.

Mike






>
> On Oct 23, 2011, at 6:25 AM, Robert McHugh <bob_mchugh_2...@yahoo.com> wrote:
>
>
>
>
>
>
>
> > The following code takes over a minute to run on my machine.  Is this expected behavior?
>
> > t = {"14-May-10 09:58:05", {"Day", "-", "MonthNameShort", "-", "YearShort", " ", "Hour", ":", "Minute", ":", "Second"}};
> > tList = Table[t, {i, 100000}];
> > a = Timing[AbsoluteTime[#] & /@ tList ;]
>
> > For reference, an operation like the following takes less than a tenth of a second. (Of course this second example needs quite a bit of modification to provide a correct answer, but it does show how fast the program can operate on a large list.)
> > t = {14, 5, 10, 9, 58, 05};
> > tList = Table[t, {i, 100000}];
> > a = Timing[( ((#[[3]] 0 + #[[2]] 30 + #[[1]]24) + #[[4]]) 60 + #[[5]]) 60 + #[[6]] & /@ tList;]
>
> > Some background: am analyzing some historical data (about 500 000 records, one data point a minute for about a year) and am making a few utilities to retrieve the data for any given time interval.   My original plan was to change the time stamp to absolute time and then use a select statement.  This step in the above example, changing the time stamps to absolute time, is the rate limiting step in the code (everything else runs in about 5 seconds).
>
> > Was wondering if someone could explain why AbsoluteTime[] is relatively slow operation and perhaps some faster operations for date and time comparisons.
>
> > Thanks

[toc] | [prev] | [next] | [standalone]


#5882

FromArmand Tamzarian <mike.honeychurch@gmail.com>
Date2011-10-26 21:45 +0000
Message-ID<j89v19$83$1@smc.vnet.net>
In reply to#5853
On Oct 25, 9:22 pm, Armand Tamzarian <mike.honeychu...@gmail.com>
wrote:
> On Oct 24, 8:13 pm, Michael Stern <nycst...@gmail.com> wrote:
>
> > All the date functions in Mathematica are extremely slow. In benchmarks I sent to Wolfram a few months ago (and some of which can be seen athttp://www.wheels.org/monkeywrench/?p=453), the date functions in Mathematica ran about 1/82 the speed of the same functions in Visual Basic for Application s. If you know the format of the dates you are converting, you may be able to roll your own date functions much faster than Wolfram's. I have encourag ed them to improve this code, but they will probably pay closer attention if others complain as well.
>
> > Michael
>
> > Sent from my iPad
>
> I'd have to agree with you.
>
> If you're doing thousands or tens of thousands of calculations daily
> that require date manipulations these are often the rate determining
> step. It seems like an example where a "super function" is a
> significant drawback. What I mean is that if you are working with a
> constant format you can, as Michael says above, write your own
> function that is between one and two orders of magnitude faster than
> the built in function. For the volumes of calculations that you might
> find yourself doing if working in a financial entity, these efficiency
> gains can be the difference between Mathematica being a competitive software
> option or not.
>
> A competitor software might have a different function for each date
> format, analogous to the roll your own function that you write,
> whereas Mathematica might have one super function that handles all format s
> etc. etc. (but slowly). For prototyping this is not likely to be a
> problem. For large scale high volumes of calculations the competitor
> wins on efficiency.
>
> Mike
>
>
>
>
>
>
>
>
>
> > On Oct 23, 2011, at 6:25 AM, Robert McHugh <bob_mchugh_2...@yahoo.com> wrote:
>
> > > The following code takes over a minute to run on my machine.  Is this expected behavior?
>
> > > t = {"14-May-10 09:58:05", {"Day", "-", "MonthNameShort", "-", "YearShort", " ", "Hour", ":", "Minute", ":", "Second"}};
> > > tList = Table[t, {i, 100000}];
> > > a = Timing[AbsoluteTime[#] & /@ tList ;]
>
> > > For reference, an operation like the following takes less than a tenth of a second. (Of course this second example needs quite a bit of modification to provide a correct answer, but it does show how fast the program can operate on a large list.)
> > > t = {14, 5, 10, 9, 58, 05};
> > > tList = Table[t, {i, 100000}];
> > > a = Timing[( ((#[[3]] 0 + #[[2]] 30 + #[[1]]24) + #[[4]]) 60 + #[[5]]) 60 + #[[6]] & /@ tList;]
>
> > > Some background: am analyzing some historical data (about 500 000 records, one data point a minute for about a year) and am making a few utilities to retrieve the data for any given time interval.   My original plan was to change the time stamp to absolute time and then use a select statement.  This step in the above example, changing the time stamps to absolute time, is the rate limiting step in the code (everything else runs in about 5 seconds).
>
> > > Was wondering if someone could explain why AbsoluteTime[] is relatively slow operation and perhaps some faster operations for date and time comparisons.
>
> > > Thanks

On a related matter, one of the frustrating things I find is lack of
packing for date lists. It would be very useful if lists of dates
could be formed into packed arrays. Each element in a date list is a
known type (Integer for the first 5 and Real for the last) so given
that fixed type format of date lists it seems to me that packing could
be implemented.

Additionally if you had time series data in which the data were the
same type you basically have a list of the form {date/time (all fixed
types), value (fixed type)} so being able to pack time series data
would improve the efficiency of these operations immensely. Currently
these operations are slow relative to other list operations due (in
part) to the lack of packing.

Mike

[toc] | [prev] | [next] | [standalone]


#5907

FromDrMajorBob <btreat1@austin.rr.com>
Date2011-10-27 10:30 +0000
Message-ID<j8bbrt$8j2$1@smc.vnet.net>
In reply to#5830
Mike,

Packing might be nice, but the last three elements of a date list do NOT  
have to be Real, the third element doesn't have to be an Integer, and the  
list isn't even required to have six elements.

Help gives no hint as to why these lists don't work:

AbsoluteTime@{2011, 10, 3, N@Pi, 2, E}

(error beep)

and

AbsoluteTime@{2011, 10, 3, Pi, 2, N@E}

(error beep)

but this one does:

AbsoluteTime@{2011, 10, 3, N@Pi, 2, N@E}

3.5266*10^9

Bobby

On Wed, 26 Oct 2011 16:39:09 -0500, Armand Tamzarian  
<mike.honeychurch@gmail.com> wrote:

> On Oct 25, 9:22 pm, Armand Tamzarian <mike.honeychu...@gmail.com>
> wrote:
>> On Oct 24, 8:13 pm, Michael Stern <nycst...@gmail.com> wrote:
>>
>> > All the date functions in Mathematica are extremely slow. In  
>> benchmarks I sent to Wolfram a few months ago (and some of which can be  
>> seen athttp://www.wheels.org/monkeywrench/?p=453), the date functions  
>> in Mathematica ran about 1/82 the speed of the same functions in Visual  
>> Basic for Application s. If you know the format of the dates you are  
>> converting, you may be able to roll your own date functions much faster  
>> than Wolfram's. I have encourag ed them to improve this code, but they  
>> will probably pay closer attention if others complain as well.
>>
>> > Michael
>>
>> > Sent from my iPad
>>
>> I'd have to agree with you.
>>
>> If you're doing thousands or tens of thousands of calculations daily
>> that require date manipulations these are often the rate determining
>> step. It seems like an example where a "super function" is a
>> significant drawback. What I mean is that if you are working with a
>> constant format you can, as Michael says above, write your own
>> function that is between one and two orders of magnitude faster than
>> the built in function. For the volumes of calculations that you might
>> find yourself doing if working in a financial entity, these efficiency
>> gains can be the difference between Mathematica being a competitive  
>> software
>> option or not.
>>
>> A competitor software might have a different function for each date
>> format, analogous to the roll your own function that you write,
>> whereas Mathematica might have one super function that handles all  
>> format s
>> etc. etc. (but slowly). For prototyping this is not likely to be a
>> problem. For large scale high volumes of calculations the competitor
>> wins on efficiency.
>>
>> Mike
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> > On Oct 23, 2011, at 6:25 AM, Robert McHugh  
>> <bob_mchugh_2...@yahoo.com> wrote:
>>
>> > > The following code takes over a minute to run on my machine.  Is  
>> this expected behavior?
>>
>> > > t = {"14-May-10 09:58:05", {"Day", "-", "MonthNameShort", "-",  
>> "YearShort", " ", "Hour", ":", "Minute", ":", "Second"}};
>> > > tList = Table[t, {i, 100000}];
>> > > a = Timing[AbsoluteTime[#] & /@ tList ;]
>>
>> > > For reference, an operation like the following takes less than a  
>> tenth of a second. (Of course this second example needs quite a bit of  
>> modification to provide a correct answer, but it does show how fast the  
>> program can operate on a large list.)
>> > > t = {14, 5, 10, 9, 58, 05};
>> > > tList = Table[t, {i, 100000}];
>> > > a = Timing[( ((#[[3]] 0 + #[[2]] 30 + #[[1]]24) + #[[4]]) 60 +  
>> #[[5]]) 60 + #[[6]] & /@ tList;]
>>
>> > > Some background: am analyzing some historical data (about 500 000  
>> records, one data point a minute for about a year) and am making a few  
>> utilities to retrieve the data for any given time interval.   My  
>> original plan was to change the time stamp to absolute time and then  
>> use a select statement.  This step in the above example, changing the  
>> time stamps to absolute time, is the rate limiting step in the code  
>> (everything else runs in about 5 seconds).
>>
>> > > Was wondering if someone could explain why AbsoluteTime[] is  
>> relatively slow operation and perhaps some faster operations for date  
>> and time comparisons.
>>
>> > > Thanks
>
> On a related matter, one of the frustrating things I find is lack of
> packing for date lists. It would be very useful if lists of dates
> could be formed into packed arrays. Each element in a date list is a
> known type (Integer for the first 5 and Real for the last) so given
> that fixed type format of date lists it seems to me that packing could
> be implemented.
>
> Additionally if you had time series data in which the data were the
> same type you basically have a list of the form {date/time (all fixed
> types), value (fixed type)} so being able to pack time series data
> would improve the efficiency of these operations immensely. Currently
> these operations are slow relative to other list operations due (in
> part) to the lack of packing.
>
> Mike
>


-- 
DrMajorBob@yahoo.com

[toc] | [prev] | [standalone]


Back to top | Article view | comp.soft-sys.math.mathematica


csiph-web