Jump to content


Photo
- - - - -

Memory files


14 replies to this topic

#1 Wim Soutendijk

Wim Soutendijk

    ProIV Guru

  • Members
  • PipPipPipPipPip
  • 211 posts
  • Gender:Male
  • Location:Netherlands

Posted 02 December 2004 - 03:49 PM

We are considering to use memory files for a couple of workfiles to speed up a time consuming calculation
we use proiv 5.5 on a windows server connected to a separate MS SQL server, and all the workfiles are in SQL server as well.
there are aprox 150 pro-iv users on that box, and we have had memory problems in the past because of the SQL layer.
How many of you use memory files?
what is the best way to create them and to delete them after processing (clear flag?)

any suggestions are welcome

Wim Soutendijk

#2 Bill Loven

Bill Loven

    Expert

  • Members
  • PipPipPipPip
  • 147 posts
  • Gender:Male
  • Location:Coppell, United States

Posted 02 December 2004 - 04:38 PM

B) Wim, I am on 5.5 also and use memory files for all transactions. We are 100% GUI and do not want to issue a rollback when the user activates the Cancel Button. Since everything is in memory, I just clear them on the way out. At last count, I have over 100 different memory files. We used to use Oracle or Pro-Isam for any work files.

Also, memory is cheap, so boost your server memory as far as you can.

HTH,

Bill

#3 Lewis Mccabe

Lewis Mccabe

    ProIV Guru

  • Members
  • PipPipPipPipPip
  • 455 posts
  • Gender:Male
  • Location:Sarasota, Florida

Posted 02 December 2004 - 04:48 PM

We use memory files extensively as well. We wrote one application which, for performance reasons, needed memory files. We read in thousands of records and process without a hitch. There was a bug a while ago which caused the last field of a memory file to be dropped. This has been fixed. Be sure you are on the latest version. Also from what I understand, memory files are set to a record length of 512 which cannot be changed. If this is no longer the case someone correct me.

Lew

#4 Lewis Mccabe

Lewis Mccabe

    ProIV Guru

  • Members
  • PipPipPipPipPip
  • 455 posts
  • Gender:Male
  • Location:Sarasota, Florida

Posted 02 December 2004 - 04:55 PM

There is no creating them except for setting the file type to MKY. We clear by both clear flag and updates as dictated by needs. The easiest method is to use a global function with the clear flag set. You don't need a primary file other than one of your memory files. Files will be cleared by virtue of the clear flag first. The LS will attempt to read the primary, error and exit. Usiing this method you can pack as many memory files into an LS as you need instead of having an LS for each file. Where you will need to clear a single or a couple of files often in a function, for performance reasons it will be better to call a local LS. Cannot use clear flag here because clear flag causes files to be cleared once per function and not per LS.

Lew

#5 Bill Loven

Bill Loven

    Expert

  • Members
  • PipPipPipPip
  • 147 posts
  • Gender:Male
  • Location:Coppell, United States

Posted 02 December 2004 - 05:46 PM

B) Lewis, you are correct about the 512 size limit. I have a couple of Oracle tables that have a record length of 1000 bytes. I have to split them in to two memory files then compine them on the write back to Oracle. It is a pain to do. ProIV is aware of this but I do not know of any impending fix.

Bill.

#6 Rob Donovan

Rob Donovan

    rob@proivrc.com

  • Admin
  • 1,640 posts
  • Gender:Male
  • Location:Spain

Posted 02 December 2004 - 08:09 PM

Hi,

Just be aware, that memory files do not return memory to the OS when you delete records from them, even if you use the Clear flag.

If you have a high user base, then this will make each kernel that is running use more memory and you could get into problems if you use them too much.

I've asked ProIV LTD on serveral occasions to 'fix' this, but it has not been done.

Rob.

#7 Wim Soutendijk

Wim Soutendijk

    ProIV Guru

  • Members
  • PipPipPipPipPip
  • 211 posts
  • Gender:Male
  • Location:Netherlands

Posted 03 December 2004 - 09:37 AM

Thank you all, I will give it a try

#8 Guest_Guest_*

Guest_Guest_*
  • Guests

Posted 03 December 2004 - 05:37 PM

Rather than using MKY files would putting your temporary pro-isam files in a ramdisk volume achieve the same performance?

#9 Joseph Bove

Joseph Bove

    ProIV Guru

  • Members
  • PipPipPipPipPip
  • 756 posts
  • Gender:Male
  • Location:Ramsey, United States

Posted 03 December 2004 - 05:51 PM

Rob,

To follow up on what you're saying:

If you have a high user base, then this will make each kernel that is running use more memory and you could get into problems if you use them too much.


I wait to appreciate the high water mark - using the following scenario

A user uses an MKY file to store 1 meg of info.
The MKY file is cleared
The user re-uses the MKY file this time to store 2 megs of info.
The MKY file is cleared again.

Is the amount of memory that is not returned to the OS 2 megs or 3 megs?

If it is 3 megs, then it would seem that the potential for system disabling memory leaks is quite high.

Thanks,

Joseph

#10 Wim Soutendijk

Wim Soutendijk

    ProIV Guru

  • Members
  • PipPipPipPipPip
  • 211 posts
  • Gender:Male
  • Location:Netherlands

Posted 04 December 2004 - 10:10 AM

I have ran some test on the development server, just loading a huge file into a memory file. In Task manager I can see memory usage increasing considerably, because of the large file that is created, but after I run the update which clears the memory file using the clear flag, Task Manager immediately shows a decrease in memory usage, almost back to the original level.
If I don't clear the file, but log out of my session, the memory seems to be returned also.
I don't see any memory leaks occuring.

#11 Rob Donovan

Rob Donovan

    rob@proivrc.com

  • Admin
  • 1,640 posts
  • Gender:Male
  • Location:Spain

Posted 05 December 2004 - 08:31 AM

Hi,

Have to admit, I've never tried it on windows, but I just did and it does seem to return most of the memory back when the clear flag is set.

The pro32srv.exe process had ~10MB of memory, I filled a memory file until about 100MB and then cleared it down. After doing this a couple of times, pro32srv.exe would not go below 45MB, even when there were no records in the memory file. So I think there is still some memory leak somewhere.

I'll have to check it again on Unix, maybe they have fixed it now, but I'm sure I tested it a couple of months back.

It would just keep eating memory and not release anything back to the OS.

Rob.

#12 Mike Nicholson

Mike Nicholson

    Expert

  • Members
  • PipPipPipPip
  • 196 posts
  • Gender:Male
  • Location:Stockholm, Sweden

Posted 08 December 2004 - 08:37 AM

Rather than using MKY files would putting your temporary pro-isam files in a ramdisk volume achieve the same performance?


We did this on a VAX machine at an old site and it worked fine with a significant performance increase. Whether it's better or worse than memory files I couldn't say though (they didn't exist at that time...)

Cheers

Mike

#13 Cleve Haynes

Cleve Haynes

    Expert

  • Members
  • PipPipPipPip
  • 172 posts
  • Gender:Male

Posted 08 December 2004 - 03:08 PM

Rather than using MKY files would putting your temporary pro-isam files in a ramdisk volume achieve the same performance?

We did this on a VAX machine at an old site and it worked fine with a significant performance increase. Whether it's better or worse than memory files I couldn't say though (they didn't exist at that time...)


We did this (use a ramdisk) on a high end Unix machine and it made no difference to peformance. The reason was that the OS was already doing significant file caching, and that Pro-Isam record locking is totally inefficient on new hardware. This is because (I think) ProIV waits too long after encountering a Pro-ISAM record lock before trying to access the record again. On a large multi-user system on modern hardware, Pro-Isam is total rubbish and you should move away from it if possible.

A memory file is much faster, since it is local to a session and does not employ any record locking.

Cleve.

#14 Guest_Guest_*

Guest_Guest_*
  • Guests

Posted 08 December 2004 - 03:31 PM

Yeah, well, Pro-ISAM wouldn't hit any record locks if only one process used each file!
However, you're right that small ProISAM files should be in filesytem cache anyway.

... Pro-Isam record locking is totally inefficient on new hardware. This is because (I think) ProIV waits too long after encountering a Pro-ISAM record lock before trying to access the record again. On a large multi-user system on modern hardware, Pro-Isam is total rubbish and you should move away from it if possible

Pro-ISAM waits can usually be tuned tolerably by appropriate spinlock settings for your hardware.. But it is rubbish :huh:

#15 Cleve Haynes

Cleve Haynes

    Expert

  • Members
  • PipPipPipPip
  • 172 posts
  • Gender:Male

Posted 08 December 2004 - 09:47 PM

Pro-ISAM waits can usually be tuned tolerably by appropriate spinlock settings for your hardware.. But it is rubbish


Depends on what your definition of "tolerable" is.

SL_SPIN is an amount specified in milliseconds. This is a very long time to wait on modern hardware. Spinlocks are somewhat useless on modern high end hardware. Why? Because the best settings for performance in /etc/isamdef are SL_SPIN = 1 & SL_NAP = 1. By the time the locked process has waited one millisecond, the process that had the semaphore has released it long ago. And you can't tune it any further!

We brought this issue up with ProIV and they agreed that spinlocks could use an "enhancement" to change the wait time to use nanoseconds. We weren't willing to pay for this "enhancement", so that's as far as it went.

Cleve.



Reply to this topic



  


0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users