Jump to content


Photo
- - - - -

Decimal Precision with SQL databases


3 replies to this topic

#1 Tyleulen

Tyleulen

    Advanced

  • Members
  • PipPipPip
  • 59 posts
  • Gender:Male

Posted 15 March 2016 - 09:12 PM

I am observing some confusing behavior using ProIV Kernel Version 7.1.58.3-PR and Postgres 9.4.

 

I am doing a calculation at full precision, 6 decimal places, I can monitor the value both before and after the file write within ProIV, and it does indeed have all 6 decimal places.  The postgres table is set to store only two decimal places.  Everything in Postgres' documentation says that if I pass more than the set number of places, it will round off the value to the correct precision.  What seems to be happening though, is that when ProIV sends the UPDATE command to Postgres, it is only passing two decimal places, not the full six.  Not only is it not passing all of the digits, it seems to be truncating the number at two decimal places, not rounding it off at all.

 

Is there any way to control this behavior?  I don't think there is any way to directly tweek the SQL command that ProIV is sending to the database, but is there perhaps a setting that will tell it to include full precision?



#2 Wim Soutendijk

Wim Soutendijk

    ProIV Guru

  • Members
  • PipPipPipPipPip
  • 211 posts
  • Gender:Male
  • Location:Netherlands

Posted 16 March 2016 - 07:48 AM

What is the external field type and numeric format in your pro4 file definition?



#3 Tyleulen

Tyleulen

    Advanced

  • Members
  • PipPipPip
  • 59 posts
  • Gender:Male

Posted 16 March 2016 - 03:17 PM

The external type is set to Number S11.2



#4 Wim Soutendijk

Wim Soutendijk

    ProIV Guru

  • Members
  • PipPipPipPipPip
  • 211 posts
  • Gender:Male
  • Location:Netherlands

Posted 17 March 2016 - 07:23 AM

If you want proiv to sent 6 decimal places to Postgress, increase that value to S11.6





Reply to this topic



  


0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users