10 replies to this topic

[Rory]

Hi

Our client is reporting thread exceptions at many points in our application (see attached)

Their IT department say their network is working fine

Would anyone have suggestions on things to check please?

Thanks
Rory

Attached Files

•  MSV_proiv_error_20080707.doc   395.5KB   245 downloads

[Rory]

Sorry should have said Kernel Version 5.5000, Revision 3.4.5 and Client 5.5 524 with Oracle 9i

[andykay]

Rory,

As, from the screen shot you provided, this looks to be a show stopper for your client, I'd first try to get your client back up and running and worry about trying to figure out the cause of the error later in one of your other environments.

What's changed since the last time the system was running without any problems? Have any of your programmers imported any changes into PROD since the last time the system worked without any problems, or perhaps made any file changes? Does your DB make use of stored procedures that might have been invalidated by a file change? If this is a show stopper that happens as soon as they log in, can you revert the PROD environment back to the last time it worked without any problems?


AK

[Rory]

Hi Andy

The client is now using the application on a standby server, this seems to be going fine (same application software & database as the production server).

Two enhancements to the application software went live on Sat, however the errors are happening in many areas as well as the modules that were updated. Plus the app is now being used without any problem on the standby server.

We don't make use of stored procedures, although (2 years ago) the client did previously add a trigger to a table without telling us which caused an Oracle problem later on. There are no triggers currently used.

The client did recently add a copy of the application to the production server for valuation purposes - we have voiced concerns about space & performance to no avail.

Northgate have the event logs and ischk output, we are hoping for a response.

Rory

[andykay]

The client is now using the application on a standby server, this seems to be going fine (same application software & database as the production server).

Two enhancements to the application software went live on Sat, however the errors are happening in many areas as well as the modules that were updated. Plus the app is now being used without any problem on the standby server.

.....What does their IT department say about their network now?

The client did recently add a copy of the application to the production server for valuation purposes - we have voiced concerns about space & performance to no avail.

.....Reading this statement makes a little hair stand out on the back of my head. I seem to remember reading a posting, perhaps on the RC...perhaps not, regarding a performance problem that was found to be able to be fixed by increasing the size of the ProIV partition. After your client had added a copy of the application, how much free space is left. Perhaps someone else can voice in if they've heard likewise about ProIV perfomance and free space.

Good Luck.

AK

[Rory]

Andy

Just to follow up on this
- we deleted the genfile
- did an isout of the remaining bootstraps
- deleted and recreated pro4 folder and isin bootstraps again
- isin genfile from dbase.out
- genned the full system
- problem seems to have gone away

Maybe genfile becomes fragmented over several years...?

We still think there are performance issues with the Oracle db & the rest of the environment but our client usually thinks Proiv is guilty until proven innocent...

Rory

[DARREN]

but our client usually thinks Proiv is guilty until proven innocent...

Isn't that allways the case with a PROIV app...in fact any application. There have been many instances where the database or network or "other software" has been at fault, but the next time there is an error it always gets reported to us first. Hey if it was easy, eveyone would be doing it right ?

[George Macken]

Hi

The problem has not gone away - it has occurred again - user got a thread error and did not not try and clear error / close client - was waiting for support and approx 30mins later the pro-iv service crashed !!!

Its hard to tell whether this user encountered initial problem or if the Thread Error encountered is sysmptom of a problem on pro-iv service caused by another user as we were not provided with info on the other sessions active.

Pro-IV 5.5, Win2003, ORACLE 9i

1) Pro-IV support say that we should check the Pro-IV File Defs versus ORACLE Table defs, to make sure the problem is not caused by a mismatch between the Pro-IV File Def and as to how the table is actually defined within ORACLE. We have been running on ORACLE 9i 4+yrs and in past if we got an error caused by the Pro-iv File Def not matching the ORACLE table the system application would fail and a pretty explicit error message would be returned. This application is regularly updated wioth new functionality and new tables, columns etc., to-date we've been pretty good at making such changes without any hiccups occurring. We have a DEV, TEST & LIVE env and gneraly any TABLE type errors would be caught in the TEST account.

DO ANY OF YOU OUT THERE HAVE A UTILITY FUNCTION WHICH RUNS THRU THE PRO-IV FIILEHDR & FILEDEF AND COMPARES THE PRO-IV DATA AGAINST THE ORACLE-TABLE WHICH HOLDS THE ORACLE DEFINITION OF THE TABLE. Checking the Alternate-Table name, the Column Names, types, size etc.,

I could probably write such a utility but its close to my holidays and i've already a lot of other work to complete. I'd appreciate any help on this item.

2) Another suggestion was to upgrade to 5.6 as it has better logging/trace we are told. However our App is developed in NATIVE (well our own DEV env based on the Native bootstraps.) Anybody else out there running 5.6 and did you work-around/avoid upgrading to VIP. Maybe upgrading tto VIP is something we'll consider doing but it cannot be done fairly imediately as a solution for this current problem.
Any thoughts on this appreciated

Thanks

George

[andykay]

George,

The fact that your client has been able to run on a standby server now for nearly 20 days now without any problems,

The client is now using the application on a standby server, this seems to be going fine (same application software & database as the production server).

and assuming you haven't made any changes since July 8th, would make me look towards Oracle partitions being the culprit. It sounds to me that some partition is getting filled up. When it does, it dumps some of the data, which makes the error go away until the partition overflows again. I remember a problem where that happened 6-7 years ago but haven't been at a site where that's happened since. Maybe, that issue was corrected many versions ago, and I'm just an 'ol geezer remembering ProIV's Golden Years .


All I'm saying is that, if after an error, the exact same transaction is processed, and many more, for many more days, are processed. Then, all of a sudden, you get another error, which it sounds like is happening to your client now...I personally wouldn't be looking at the ProIV code. If it was an issue of bad data/code causing an error like division by zero, then it'd happen every time. But if business resumes to normal for a period of time, all awhile accessing, writing to, and reading the same files...I'd be looking some other place than the ProIV code being the culprit where file size mattered once upon a time (Ie. Oracle). Would it be possible for you to write a routine in one of your lower environments to write a million contracts to the system to see if at some point the system gets overloaded?

JMO,

AK

[Chris Mackenzie]

George,

The fact that your client has been able to run on a standby server now for nearly 20 days now without any problems,

The client is now using the application on a standby server, this seems to be going fine (same application software & database as the production server).

and assuming you haven't made any changes since July 8th, would make me look towards Oracle partitions being the culprit. It sounds to me that some partition is getting filled up.

While I think Andy is likely onto something, if it were Oracle space issue then it would
be reported in the alert log. I have to assume that's been checked. As will disk
space/integrity. How about server memory usage - is that growing before these thread exceptions?

[Rory]

We made a change to cause a 1 second delay in a function that links all our screens and have been monitoring the system since then. No errors have been reported. I'm not sure if this indicates that ProIV needs the breather or the other components (network, db etc.) do.
Has anyone come across this before?

There is now a team onsite monitoring db & server performance, we will turn off the 1 second delay at some point to see what the monitoring shows up.