Path: csiph.com!optima2.xanadu-bbs.net!xanadu-bbs.net!feeder.erje.net!1.eu.feeder.erje.net!eternal-september.org!feeder.eternal-september.org!mx02.eternal-september.org!.POSTED!not-for-mail From: Eric Sosman Newsgroups: comp.lang.java.programmer Subject: Re: method micro benachmarking with reflection Date: Thu, 30 Jul 2015 14:20:58 -0400 Organization: A noiseless patient Spider Lines: 31 Message-ID: References: <7c2ab94b-ee4e-41c9-ae71-3896210e80b7@googlegroups.com> <72344bf7-bb09-45b2-9e7f-723576fbb639@googlegroups.com> <9c2e6e13-8d02-4bc6-ba02-91d9cbf4123b@googlegroups.com> Mime-Version: 1.0 Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit Injection-Date: Thu, 30 Jul 2015 18:19:29 +0000 (UTC) Injection-Info: mx02.eternal-september.org; posting-host="858a9d65d95f4ea4d355bd116aa2cdbf"; logging-data="26030"; mail-complaints-to="abuse@eternal-september.org"; posting-account="U2FsdGVkX19tChBg/R/PUp9lqPH0hx1M" User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.1.0 In-Reply-To: <9c2e6e13-8d02-4bc6-ba02-91d9cbf4123b@googlegroups.com> Cancel-Lock: sha1:J0iZ1U/SJQbTVVN/TP9GyMAvNK4= Xref: csiph.com comp.lang.java.programmer:32215 On 7/30/2015 1:30 PM, e.d.programmer@gmail.com wrote: > On Thursday, July 30, 2015 at 11:42:29 AM UTC-4, Eric Sosman wrote: >> One of the articles I linked to suggests doing "several" >> times that many loop executions ... >> >> ... in the *warmup* phase, before serious measurements. > > Right. It helps if you call the method before getting the initial timestamp to exclude the one time (classloader, public class variable creation) stuff. The point (and a repeated warning in the links provided) is that "call the method" is not enough of a warmup. You must - Call the method enough times to be sure that the JIT has finished converting it to native code, - Call *exactly* the same methods during the timing phase as you did during the warmup, with exactly the same classes loaded and so on, lest the JIT change its mind, - Check that nothing unexpected happened during the timing phase. > The 10000 was a random number. It depends how long it takes to run the method once, and how long you want to spend testing it. If 10000 calls only takes a few minutes you could do 100000 calls to be more accurate. If 10000 calls takes 10 hours, do 100000 calls if you have a week to test... If 10000 calls takes a few minutes, it seems to me that we're no longer in the realm of "micro benachmarking [sic]." Microbenchmarks are rather different from tests of system capacity, throughput, or latency. All are difficult, all present challenges of their very own. -- esosman@comcast-dot-net.invalid "Don't be afraid of work. Make work afraid of you." -- TLM