Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.databases.ms-sqlserver > #2121

Re: Strage difference between precision in literals

Show all headers | View raw

Anton Shepelev (anton.txt@gmail.com) writes:
> Hello, all
> 
> Can you please tell me why this query:
> 
>    SELECT     POWER(0.5, 2), POWER(0.50, 2)
>    Returns    0.3            0.25
>    Instead of 0.25           0.25
> 

All say after me: "SQL Server is designed for maximum confusion!".

I know how it feels, I've run into exactly this thing myself. And
I thought, what!? Then I read the manual and learnt this:

The retun type of power() is the type of the first argument. The 0.3
has the type decimal(1, 1), and so the return type is numeric(1,1)
and you get 0.3. Intuitive? Maybe not. But this is what the manual
says.

By the way, this is how you can tell the type of a numeric literal:
DECLARE @s sql_variant = 0.3
SELECT sql_variant_property(@s, 'Basetype'), 
       sql_variant_property(@s, 'Precision'), 
       sql_variant_property(@s, 'Scale')

To avoid such surpises, say something like:

  SELECT cast(power(cast @val as float) as decimal(10,3))

Back to comp.databases.ms-sqlserver | Previous | Next — Previous in thread | Next in thread | Find similar

Thread

Strage difference between precision in literals Anton Shepelev <anton.txt@gmail.com> - 2022-04-23 19:04 +0300
  Re: Strage difference between precision in literals Erland Sommarskog <esquel@sommarskog.se> - 2022-04-25 21:58 +0200
    Re: Strage difference between precision in literals Anton Shepelev <anton.txt@g{oogle}mail.com> - 2022-04-26 13:21 +0300
      Re: Strage difference between precision in literals Erland Sommarskog <esquel@sommarskog.se> - 2022-04-26 20:26 +0200

csiph-web