Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.programming > #1308

Re: Precision on size int in C...

Show all headers | View raw

On 2/8/2012 3:48 PM, Chad wrote:
> The output of 2 to the 30 for a signed int in C on my 32 bit machine
> is:
>
> 1073741824
>
> The question is how does my computer get 10 digits of precision?
>
> I thought it might be something like
>
> 2^K = 10^N
> N = K Log (2) //log is base 10
>
> If I let 1 bit for the sign, then  K = 31 bits. Plugging 31 into the
> above formula yields N roughly equals 9.33.
>
> Now if I let K = 32, N is about 9.63.
>
> In either case, it seems I'm coming out to 9 digits of precision.

I think "precision" is not quite the right way to think about this,
because in signed int each integer is either exactly representable or
not representable at all.

A result like 9.33 means that all 9 digit numbers are representable, and
so are some of the smaller 10 digit numbers. 2**30 is one of them.

Consider an easier case, 4 bit unsigned numbers. Applying the
calculation with K=4 gives a number slightly over 1.2. All the one
decimal digit numbers are representable in 4 bits, and so are the
smallest two decimal digit numbers, 10 through 15.

Patricia

Back to comp.programming | Previous | Next — Previous in thread | Find similar

Thread

Precision on size int in C... Chad <cdalten@gmail.com> - 2012-02-08 15:48 -0800
  Re: Precision on size int in C... Ben Bacarisse <ben.usenet@bsb.me.uk> - 2012-02-09 02:30 +0000
  Re: Precision on size int in C... Patricia Shanahan <pats@acm.org> - 2012-02-08 18:32 -0800

csiph-web