Jump to content


Photo
- - - - -

Bootstrap files & read only mode


11 replies to this topic

#1 Rob Donovan

Rob Donovan

    rob@proivrc.com

  • Admin
  • 1,640 posts
  • Gender:Male
  • Location:Spain

Posted 16 May 2000 - 12:31 PM

Hi,

I have been told (by PROIV LTD) that if you put the bootstrap files into read only mode (i.e. -r--r--r-- in UNIX) then PROIV will not use record locking methods on these files and loading of functions is quicker.

I realise that you cannot then do imports/gens/mods to functions as that was stated too.

Has anyone actually done this, and not seen any problems or have you seen performance gains?

Thanks,

Rob Donovan

#2 Richard Bassett

Richard Bassett

    ProIV Guru

  • Members
  • PipPipPipPipPip
  • 696 posts
  • Location:Rural France

Posted 16 May 2000 - 07:55 PM

I was also told in the past you could do this.

I doubt that it's a performance issue - ProIV doesn't do any record locking in lookup mode anyway. I'd be surprised if you saw a performance gain in simple runtime execution but please tell us all if you do!!

I always assumed the reason was so that people could write-protect the bootstraps from unscheduled or malicious changes to their application software. Originally there may have been a problem doing so because ProIV insisted on opening the files in read-write mode, probably now it tries that and then falls back on read-only mode if it fails.

Note that this is just (educated) speculation!
Nothing's as simple as you think

#3 Rob Donovan

Rob Donovan

    rob@proivrc.com

  • Admin
  • 1,640 posts
  • Gender:Male
  • Location:Spain

Posted 16 May 2000 - 08:10 PM

Hi,

What I have been told by PROIV LTD is that....

If you kill iscollect & remove shared memory and then start PROIV any files accessed with -r--r--r-- will not be put into shared memory. Even if the file is open in Lookup mode PROIV still holds an entry in Shared memory. This supposedly speeds things up, I haven't tried it yet though and I guess it would only help on a high volume system.

Rob

#4 Peter Davies

Peter Davies

    Advanced

  • Members
  • PipPipPip
  • 96 posts
  • Gender:Male
  • Location:Bangkok, Thailand

Posted 17 May 2000 - 09:12 AM

Maybe Dr PROIV can confirm this.

When you read a pro-isam record, a 'currency' is created. It is currencies that you see when you do an isview. It is this same currency that is created for record locking.

I've heard that a currency is not created on a read-only file. Sorry I can't give you 100% confirmation. Let's hope the doctor can confirm.

Even if the currency is not created, this will effect memory and not I/O. A more significant change will occur if you control the I/O.

Some operating systems have a ramdrive facility. The contents of directories/files can be loaded into memory which greatly reduces I/O.

The bootstrap files do represent I/O. An application with a large number of Global functions will have more bootstrap I/O. In screens this is OK but in updates processing high volumes of data, this I/O will impact performance. Reduce the number of globals in these updates to decrease the I/O.

As a minimum, the bootstraps should be stored on a seperate disk to the data. If possible, GENFILE, MSGF and FUNCTDEF should be stored in memory.

The caching options of the O/S will also impact the amount of real I/O as opposed to cache I/O.

Anyone else help ??

#5 Rob Donovan

Rob Donovan

    rob@proivrc.com

  • Admin
  • 1,640 posts
  • Gender:Male
  • Location:Spain

Posted 17 May 2000 - 09:29 AM

Hi,

We currently run about 1 - 1.5 Million functions per day. About 1/4 of these are global functions.

It would be nice if PROIV would cache GENFILE records for global functions, in which case calling a global function would then be as quick as an LSCALL. Currently it is impractical to call a global function 1000's of times in an update.

If global functions were cached in some way (just within the main running function) it would help.

The disk set that our bootstraps are on is in a seperate disk cabinet system (HRZ system) and the main system has GigaBytes of memory so the functions should be in the disk cache, I think.

Rob Donovan

#6 Guest_Doctor PRO IV_*

Guest_Doctor PRO IV_*
  • Guests

Posted 17 May 2000 - 09:32 AM

The benefits arise out of fact that since we know, for
certain, that no one else can be changing the structure of
the bootstrapfile, then we do not have employ the same level of protection when it comes to simultaneous access to the file by more than one user. We can allow more concurrency between processes contending for the same files such as functdef, vardef and genfile.

The benefits are much more pronounced on a multi-processor
system using spinlocks.

I do appreciate that having read-only bootstraps can cause
operational difficulties but if these can be overcome, then
the benefits may be worthwhile.

#7 Rob Fantini

Rob Fantini

    ProIV Guru

  • Members
  • PipPipPipPipPip
  • 267 posts
  • Gender:Male
  • Location:Haverhill, United States

Posted 17 May 2000 - 01:39 PM

We use DPT caching controllers.

There is a system analysis tool for most unix systems called 'sarcheck' . A demo can be had at www.sarcheck.com .

I typically notice access times of < 2ms on our cached drives and 14-20 on our non-cached drives.

Sarcheck is great for making suggestions on tuning unix systems.

#8 Richard Bassett

Richard Bassett

    ProIV Guru

  • Members
  • PipPipPipPipPip
  • 696 posts
  • Location:Rural France

Posted 17 May 2000 - 06:24 PM

OK, so it's a Pro-ISAM-specific optimization.

I guess what they're saying is that provided the file is read-only they don't have to worry about reader-processes encountering half-updated blocks and so on and they can dispense with the synchronization which protects the internal consistency of the file. It makes sense that this helps most on a multiprocessor as it'd enable more actual simultaneous processing.

If you're on Unix, you're sure bootstrap access is the source of your performance problem and the relevant bootstrap files are pretty small compared to available memory then you might investigate locking the files in memory using mmap.

I think mmap is supposed to 'map' the disk file to a block of shared memory and provide faster access to it by bypassing the usual Unix filesystem code.

CAUTION: I have never tried to achieve this, I don't know how difficult it is or indeed if it is unsafe for some reason with Pro-ISAM. Also I don't have a Unix system to hand to check if my recollection of mmap is faulty.

Perhaps Dr. PROIV might like to comment.
Nothing's as simple as you think

#9 Guest_Doctor PRO IV_*

Guest_Doctor PRO IV_*
  • Guests

Posted 17 May 2000 - 06:41 PM

Yes, Richard - It is a PRO-ISAM specific optimisation.

mmap should be used with extreme caution.

The semantics differ between different versions of Unix and the last time I looked at the Linux version it was definitely broken.

A preferable technique which is just as applicable to VMS is to use a RAM disk.

Linux and VMS definitely support such a feature and other versions of Unix has varying levels of support for such a feature.

Rob's interest in function caching has been noted.

#10 Guest_Doctor PRO IV_*

Guest_Doctor PRO IV_*
  • Guests

Posted 18 May 2000 - 09:52 AM

Please note that this optimisation is not strictly for the
bootstraps.

The first sixteen files opened which have read-only
permissions will be optimised in this way.

You could consider using it for static code files.

#11 Rob Fantini

Rob Fantini

    ProIV Guru

  • Members
  • PipPipPipPipPip
  • 267 posts
  • Gender:Male
  • Location:Haverhill, United States

Posted 18 May 2000 - 07:33 PM

What if the bootstrap files had rw-r-r permissions, with rw for developers only, and 'r' for regular users. Would regular users get a potential preformance boost?

#12 Rob Donovan

Rob Donovan

    rob@proivrc.com

  • Admin
  • 1,640 posts
  • Gender:Male
  • Location:Spain

Posted 23 May 2000 - 12:08 PM

Hi,

I got the following results from testing global functions/read only mode/Mem Disk & spinlocks......

All test cases use 1 LS to read the same record in a for loop 100,000 times. (ie 100,000 acceses of the same record)

All times are averaged over 4 runs.

Test 1:
LSCALL locally in the function

33 seconds

Test 2:
Replace the LSCALL with a GLOBAL_LSCALL

420 seconds

Test 3:
GLOBAL_LSCALL with the 'read only' bootstraps

370 seconds

Test 4:
GLOBAL_LSCALL with the bootstraps on a RAM (memory ) disk with 'writeable' bootstraps

400 seconds

Test 5:
GLOBAL_LSCALL with SPINLOCKS enabled, no ramdisk, writeable bootstraps

390 seconds

I guess that the RAMDISK does not really help as all the records will be cached in memory after the first read anyway.

It looks like the 'read only' bootstraps speeds up intensive function accessing , but I guess caching the GLOBAL LSCALLS would make them as quick as the LSCALL.

Rob Donovan



Reply to this topic



  


0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users