18 replies to this topic

[Rob Fantini]

Hello

 

we are committed to using  pro4 Version : 4.6000  Revision : 2.1.3 for the foreseeable future.

 

use use pro4 for  order entry ,  production reports, billing , accounting and a lot more.

 We've been in business since 1902  and it looks like we'll be using pro4 another 50 years .  we are looking to make pro4 as reliable as possible.

 

 

 My question is:

 

which file type is more reliable / resistant to  corruption   proisam or c-isam ?

 

 

 

 

 

thanks for reading.

 

 

[Rob Donovan]

Hi Rob,

 

After using C-ISAM and ProISAM for many years, I would say that C-ISAM is the better.

 

The main advantage, is that the data is separate from the index, and its normally the index that gets corrupted. Having the data part of the file separately gives you more options and makes it easier to recover the file in the event of any disasters.

 

There is very little that can corrupt either file types though, apart from a hard shutdown, from what I've seen, as long as you keep up good practices and keep your system in shape.

 

So that includes things like:

1) Any time there is a machine crash of any type, the system should be taken offline and ischk'ed / BCheked.

2) Keep watch of file sizes etc for ProISAM limits.

3) Watch out for things like Clr Flags on files that are accessed simultaneously (this can corrupt the file, if other people are accessing it and it gets cleared)

4) Backups. Make sure everyone is off the system, and no file is 'open' while taking the backup (Or some kind of shadow/split system that can backup 'live' files)

5) Not necessarily going to help with corruption as such, but is generally a good idea.... Doing defrags and isout/isin's to keep the data in a good shape periodically. This would also involve doing ischk/bcheck first to check everything is good first. If there is some weird corruption somewhere, then its best to catch it as early as possible so that it doesn't get worse and corrupt more, and more data can be 'saved'.

 

Following good procedures like these means I've only ever seen file corruption if we had a hard system crash. As in an unexpected reboot while users were accessing the files or some kind of disk failures. (Although there have been a few instances of ProISAM corruption in the past due to bugs in the kernel, but they are normally around 'new kernels' when I've upgraded systems.)

 

HTHs, Rob.

[Rob Fantini]

Hi Rob

 

 Thank you for the very helpful answer.       Usually we'll get a corrupt file when there are major virtualization host  upgrades [ we use a proxmox cluster and ceph storage along with zfs for backup].  Sometime there are bugs , sometimes there are learning experiences ;-)  like mis configurations.    despite being careful there is always so much more to learn.   Our worst issue was in June when I was setting up our stack of 3 managed switches - and tested fail over.  long story but next time I'll take pro4 off line when doing more tests.  

 

I have found c-isam has been more reliable during these tests and upgrades.   I  was considering converting the 100+  proisam files .  Now after  we will.

 

Question - say a c-isam file does get corrupted -  how does one rebuild the index?

 

 

Thank you  and hope all is going well , Rob.

[Rob Donovan]

Well, with CISAM you have 2 choices.

 

Obviously work on a copy, but try Bcheck first, it can rebuild corrupt indexes. But after it, you need to check that the file then has the appropriate number of records etc in it, as it could get it wrong.

 

If it's very badly damaged or you find too much data missing or corrupt, then you could try 'homemade' scripts/functions that read the pure .dat file as 'text' and then add it back into a new clean CISAM file. This is a method I've used before to save data.

 

Other than that, its a very 'see what has happened' type of thing.

 

As for converting to CISAM, remember, CISAM is fixed record length, so will take up more disk space per file and generate more IO and locks on the system. Any differences would be hard to judge off hand, and hugely depends on your data, number of users on your system etc etc.

 

Also on a programming side remember that some problems can occur with just swapping to CISAM as there are a few differences with CISAM. Specifically with ProISAM a 'holder/lock' records is actually added when the file is in 'A' mode between 'BeforeRead' and 'Before Write' (and therefor no 'lock' of that record), but with CISAM nothing is added until the BeforeWrite point. This can cause problems sometimes depending on how your system is working with BeforeWrite failing because of 'duplicate' records. Normally happens with AutoSeq files, because things happen in-between the GetSeq and the write. But again, hugely depends on your system and how it works.

 

Life is good, currently not using P4 much as I'm trying to develop and release a Video Game in Unity and C# that we are working on (https://www.facebook.com/rokabiumgames)... so that's fun! 

 

Hope all goes well with converting to CISAM, in the end it should be good for your system.

[Rob Fantini]

Hello again Rob D.

 

 

do you have a suggestion for which 'STORAGE FORMAT' may be best to use?

 

 

PS:

way back I had done extensive testing on cisam storage format - to see which type used less storage.   i'll continue to search this forum for those results.

 

more important then which data type takes less storage is which are more reliable and potentially readable by external applications.

[Rob Donovan]

Obviously, ALPHA for strings.

 

As for numerics, I've used a mixture, can't quite remember why TBH, sorry... I think it could have been in the past there was some bugs with them.

 

Think small numerics under 4.0 we used 'ZONED' which saves them as strings, and then COMP3 for anything else, because COMP3 had a problem with smaller numbers if I recall correctly.

[Rob Fantini]

thanks for the fast response.   I've been going over the manual a few times and was going to edit my post to add this:

"C-ISAM can store data on disk in several ways; any of these can be read from or written to using a PROIV
application. The C-ISAM data types and the corresponding entries at the PROIV file definition prompt
EXTERNAL TYPE is as follows:"

C-ISAM Type PROIV Type Comments

CHAR       ALPHA     ASCII character data
INT        PCOMP1    Two-byte integer
LONG       PCOMP2    Four-byte integer
DECIMAL    C-DEC     Decimal Format, storage format required
FLOAT      C-FLT     Single Precision floating point
DOUBLE     C-DBL     Double Precision floating point

Field Considerations
DECIMAL Type Fields
DECIMAL types are stored in the proprietary C-ISAM format. They require a standard PROIV STORAGE
FORMAT of the form:
{S} i . f
where the optional S indicates a signed number, i is a number of integer digits, and f is a number of
fractional digits (for example 5.2 or S3.1). The size of a DECIMAL data type field can be calculated using
the following formula:
number of bytes = INT (i/2 + 1/2) + INT (f/2 + 1/2) + 1
See the section, Sizing DECIMALTYPE Numbers, in your C-ISAM Programmers Manual for further
details.
FLOAT and DOUBLE Type Fields
FLOAT and DOUBLE type fields are defined by C-ISAM with respect to the C programming language for
the host machine. Typically, FLOAT types are 4 bytes and DOUBLE 8 bytes.

that is from PROIV Env. Guide.

Edited by Rob Fantini, 25 March 2018 - 02:41 PM.

[Rob Fantini]

more: " All other PROIV external types can be used with C-ISAM files, but non-PROIV applications may need
additional programming to convert them."

 

 More important then allowing external  programs having access to data is pro4 data stability.

 

For now I'll use  ALPHA and  C-FLT

Edited by Rob Fantini, 25 March 2018 - 02:51 PM.

[Rob Donovan]

I would think they are all fairly similar for 'stability' TBH.

 

We used COMP3 from the start (starting back in 1994) probably for size, since the only reason we went to CISAM then was the size limitations on ProISAM back then.

 

We had external C++ code that read our C-ISAM files after a while, so we just got them to decode the COMP3s.

[Rob Fantini]

 Rob - thank you for the info.  we'll go with   ALPHA and COMP3

[Rob Fantini]

ALPHA and C-FLT we settled on . because we have been using c-flt for a while in our invoicing files.

[Richard Bassett]

Hi Rob,

 

To add 0.02:

 

C-ISAM certainly is more dependable.  Using Pro-ISAM only for ProIV's own bootstrap files is a rational objective.  And you might be able to exploit the more-well-known nature of C-ISAM to get SQL access direct to your data, if indeed you're not doing that already.   (Although I'm not sure if there are 'free' solutions out there for that)

 

ProIV actually had support for C-ISAM with variable length records (external file type CISAMV), however, in C-ISAM that worked by putting data into the index (.idx) and so compromised the reliability/recoverability.  In any case I think that support might have been a specially licenced option.

 

It's been a long time since I looked at this stuff but I am mildly concerned about your choice of C-FLT as a data type, expecially for financial data.

 

C-FLT directly represents the C programming 'float' type, which is a 'low' precision binary floating point type.

Because it's a binary floating point type it doesn't accurately represent decimal fractions (e.g. cents)

The maximum precision is likely to be 9 decimal digits total (maybe OK, but wouldn't be in banking etc.)

The storage representation will be machine-specific, whereas C-ISAM can store data independently of chip architectures.

 

For these reasons, I think C-ISAM has always suggested/recommended using the DECIMAL (C-DEC) type and, personally, I would recommend that too.

 

Cheers,  Richard

 

PS.  I noticed something seems slightly wrong with some links on your own main www page - stuff is linked to localhost:10080 ?

[Rob Fantini]

Hello Richard -   

 Thank you for the response  -  I will switch over to C-DEC .     

 

 

 

PS:

also on the bad link you mentioned  - is that on forum profile or our company web page?

[Richard Bassett]

I meant the .com landing page for your company, I just hesitated to namecheck it although you probably mentioned it in the past anyway

[Rob Fantini]

i'll check that web page - thanks

 

regarding C-DEC  vs C-FLT  .  I just tried to use C-DEC  on a new file.   

 

using an update i posted  proisam file to cisam  , and most of the C-DEC fields do not display data .    I'll continue to debug...