14 replies to this topic

[Gary Wilhelm]

Hiya gang

Our system is a 24/7 that runs on Linux. For the last few nights, iscollect has had open files and hasn't wanted to release them. How do I force iscollect to release open files without interupting operation? Here is the report that I get from my nightly backups.

COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
iscollect 27681 root cwd DIR 3,1 12288 656612 /usr/pro/job
iscollect 27681 root 3u REG 3,1 2549760 132031 /usr/pro/msgf.pro
iscollect 27681 root 5u REG 3,1 924672 132007 /usr/pro/functdef.pro
iscollect 27681 root 6u REG 3,1 26938368 132012 /usr/pro/genfile.pro
iscollect 27681 root 7u REG 3,1 138240 132045 /usr/pro/security.pro
iscollect 27681 root 9u REG 3,1 4293120 640291 /usr/pro/job/pr/ymempls.pro
iscollect 27681 root 10u REG 3,1 27763200 640289 /usr/pro/job/pr/ymehist.pro

Thanx,
-Gary

[Joseph Bove]

Gary,

Since it is during your backup routine: I think that the easiest answer is to just kill iscollect before running the backup.

Specifically:

kill `ps -edaf | grep iscollect | grep -v grep | awk '{print $2}'`

Obviously, I'd test that a test box first!

hth,

Joseph

[Richard Bassett]

Hmm,

[Caveat: Long time since I thought I was on top of any of this stuff]

Not sure killing iscollect can work, especially in a 24x7 system.
AFAIR, every new ProIV kernel process that starts up can kick off a new iscollect if there doesn't seem to be one active.
(Actually a new kernel may always kick off a new iscollect anyway and the redundant iscollect may just die if there is already an active iscollect in existence - I can't remember)

A more important question is why iscollect has any ProISAM file open - it certainly doesn't need to as far as I remember.

They could just be spurious open files that the active iscollect "inherited" from whatever kernel process originally "spawned" it. That should be something ProIV could fix maybe?

[Gary Wilhelm]

The problem is NOT an ongoing problem. It's been running fine for years. It just came up a few days ago and hasn't cleared itself, so killing iscollect doesn't need to be a permanent option before backup. I just need to know if there is a way to fix it without having to force everyone off of the system and do a reboot.

Thankx,
-Gary

[Gary Wilhelm]

What would happen if I killed iscollect while everyone is on the system?

[Richard Bassett]

Pretty sure iscollect doesnt open proisam files and I also wouldnt expect the kernel to have any open at the time it is started

Yeah, I did wonder about that. The list of files looks just like what a kernel would have open though. I suppose it's conceivable a kernel can spawn an iscollect other than when it starts? Maybe on certain ProISAM operations to cater for the situation where iscollect crashes and no new kernels are started? [This is pure speculation on my part].

What would happen if I killed iscollect while everyone is on the system?

Well, nothing immediately bad happens. What iscollect basically does is look for record locks held by kernel processes that have died and cleans them up. So you wouldn't have that service temporarily.

If you used kill -9 then I don't know if there's any possibility of iscollect leaving the shared memory segment in an inconsistent state if you happened to catch it while it was modifying the segment. It's probably pretty unlikely.

It would be an interesting experiment to do this while kernels are active (doing ProISAM work) and without starting any new kernel - to see if any iscollect process was automatically restarted.

Incidentally Gary I'm a bit confused about one thing - if there are kernels running while you are doing backup then why aren't their open files a problem?

[Gary Wilhelm]

Incidentally Gary I'm a bit confused about one thing - if there are kernels running while you are doing backup then why aren't their open files a problem?

There are standing instructions to not use the system for about 45 minutes starting at midnight.

That gives me an idea. I could write a script that kills the process just before backup starts. One time only of course. Then just need to make sure iscollect restarts.

[Lewis Mccabe]

Iscollect will start automatically on 1st pro process if it is not already running. You will not need to restart it after your backup.

Edited by Lewis Mccabe, 18 January 2005 - 04:57 PM.

[Gary Wilhelm]

This is a normal backup result file

COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
iscollect 2101 root cwd DIR 3,1 12288 656612 /usr/pro/job

[Joseph Bove]

What would happen if I killed iscollect while everyone is on the system?

From what I've seen, you lose all of your current record locks. This has the potential to be problematic and may also corrupt files.

I try to never kill iscollect on a live system - unless iscollect is running at 100% of CPU.

hth,

Joseph

[Rob Donovan]

Hi,

As others have said, Iscollect just clears up rogue record locks, it does not open ProISAM files.

When you say 'they are told not to do anything for 45 mintes'... what do you mean? Do they logout of ProIV / Unix?

You should make sure that they are out of the system, and you should disable logins during backups of ProISAM files, otherwise you could be backing up a ProISAM file that is being modified and that would mean that the file backed up would actually be corrupt on the backup, and of no help to you for restores.

The only way to make sure that there are no open ProISAM files, is to make sure that iscollect is running, and make sure that there are no 'pro ' processes running. (using the ps command)

You can find specific processes with open ProISAM files using the ISVIEW ProIV utility.

Rob D.

[Gary Wilhelm]

Yes logins are disabled during backup
Yes every precaution is done to make sure files are not backed up that are open.

Please refer to the first question. Iscollect has done something that is strange. We all agree on that. It looks like it has open pro files when it isn't supposed to do that by design. I don't need help with my backups, they work just fine. The question is concerning iscollect and killing it. I have solved the symptom on this particular occassion. I wonder if anyone knows what caused iscollect to have these open files. Does iscollect take over users when they exit and try to close open files?

[Richard Bassett]

I repeat my speculation that your iscollect had .pro files open because it happened to be spawned by a kernel under some "unusual" circumstance.

I repeat my suggestion that you could do an experiment to kill iscollect while kernels are running and then wait to see if an iscollect reappears even if you start no new kernel processes. Obviously that doen't have to be on a production system.

HTH.