22 replies to this topic

[scminter]

Attached is a trace file generated right before my Pro32srv process abnormally ended.

Event ID: 7031

The PROIV NT Server service terminated unexpectedly. It has done this 1 time(s). The following corrective action will be taken in 60000 milliseconds: Restart the service.


This has been occurring every day a couple of times. I'm new to Pro-IV on windows. I did trace the error anc believe I have attached the trace file.

Any help would be appreciated.

Also, any tips on how to read the trace files would be appreciated (teach the man to fish...).

My pro-iv kernel version is 4.6 rev 5.3.4 (glovia) on Windows Server 2003 Standard Edition Service Pack 1.

[scminter]

The attachment.

Attached Files

•  91c.zip   296.25KB   116 downloads

[Joseph Bove]

scminter,

Some tips.

First and foremost: Always try to stop the trace immediately after generating the error. The less trace to read through the better.

Search the trace for error or message or @RFUNCT or 366.

If you have specific @RFUNCT error messages, then the system level error will occur slightly before the capturing of the @RFUNCT message in the trace.

If you know what function you've errored in, you can almost always throw out all of the trace prior to that function.

Also, you can compare a good trace to a bad one to see what the differences are. This is quite lengthy though.

Finally, don't hesitate to get ProIV involved!

...

It looks like you've got 2 references for @rfunct in your trace. One is one a 366 error the other on an 18 error. Not knowing the app or the location of the error - Is this meaningful to you?

018 - ERROR IN WRITING FILE - DRIVER

hth,

Joseph

[andykay]

DRIVER is a GLOVIA file that is used as the Driving File for the Outermost Loop of a LU. It should only have a NULL key (only 1 record in the entire file) and never write to it.

Look for a function that has DRIVER in "A" or "B" mode and remove that reference.

HTH,

AK

Edited by andykay, 31 May 2006 - 11:55 PM.

[scminter]

The Pro32Srv service abnormally ended at 8:19:21AM on 30May06. Once the server process crashed, I stopped tracing so that when it started up again there would be no further logs.

[Lewis Mccabe]

Have you recently modified any programs?

[scminter]

With your replies I searched for a function the tries to write to Driver in Add mode. I found the function and executed it. The system crashed.

The function = GLOVIA. Apparently it tries to read the file DRIVER, then tries to add a null record if the read failed. I don't know why the read failed since the record is there. I believe this might actually be an Oracle error where it fails on the read of a null key. I had had an occurrance of an ORA-24364 in the trace file.

[scminter]

I have not modified any programs. However, it appears my predecessor changed the function GLOVIA, but never regenned it. As part of another process GLOVIA had me regen all functions. So, whatever change he made has now been genned. I will have to ask Glovia for the original function to compare and see what he did.

[andykay]

scminter,

GLOVIA is the very first screen that you see after loging into ProIV. It is used to verify the presence of a few files that are commonly used throughout Glovia, and in fact are required. One of these files is the DRIVER file. The GLOVIA screen function first attempts to see if there is a valid record already and if not, will attempt to add it so that no other functions that may use it later, from within Glovia, fails.

So, in fact I do stand corrected . The ONLY screen that should have DRIVER in "A" mode is in fact GLOVIA, as it will create a NULL record if it is not present when you first get into Glovia because of its importance as the driving file in other functions.

If Glovia is attempting to write to this file, then have you verified that the NULL record is in fact there? If it is, perhaps the file is corrupt. Try locating an Update or a Report that may use this file and run it. Place a UMSG in the ARNE to verify that you are able to read the file, as well as ARE and see which UMSG you get. The final thing you could try, if you can't wait for Glovia to send you the GLOVIA screen, is Delete the bad record in DRIVER and then re-add a NULL record back into DRIVER.

Good Luck,

AK

[scminter]

Thanks.

What I did was to lsexit and never try to add the record.

In the trace file you can see the read fails and then pro-iv tries to insert the Null (string) again. This fails because the record does exist.

=== Pasted from the trace file ===
081825.391-OraCmdParse: SQL STATEMENT(length = 50) =
081825.391- SELECT NULL_KEY FROM DRIVER WHERE NULL_KEY=:K00000
081825.391-qgRollBackException: exception c0000005 Address 604af6fb occured in qgSqlFRead, SQL filename 'DRIVER' -- see Event Log
081825.391-OraRollback: Rollback OK - rc = 0
081825.391-qgSqlFInsert: pTib 07BE6DB0, fAccess 0(INIT)
081825.391- Current ProIV function GLOVIA ls 6 fldno 16, curfile 11, 'DRIVER'
081825.391-qgPrmPreRefs: ColumnReference ptr = 00B42944
081825.391-qgPrmCbPrepare: Prepare Parameter Control Block
081825.391-OraSqldaVInit: Alloc heap(256), pSqlda->hHeap = 07BE4060
081825.391-OraSqldaVInit: Alloc PVars: pSqlda->pVars = 03D2A264, SIZE = 1040
081825.391-OraSqldaReset: pSqlda = 067E0854 fParse = 3
081825.391-OraCmdParse: SQL STATEMENT(length = 46) =
081825.391- INSERT INTO DRIVER(NULL_KEY) VALUES (:V00000)
081825.391-qgRollBackException: exception c0000005 Address 604af6fb occured in qgSqlFInsert, SQL filename 'DRIVER' -- see Event Log
081825.391-OraRollback: Rollback OK - rc = 0
081825.391-sysfwrt: exit, stat -999999, errno/fnl.filerrno 0/-999999
081825.391-guiHint: 03=UMSG 'null'
081825.391- 1b{ ! 000903000000
081825.391-sysCRTWrite : len 9: 1b{ ! 000903000000
081825.391-ClientWrite: buflen 9:
081825.391- 1b{ ! 000903000000
081825.391-sockSend: sock d994, buflen 9:
081825.391- 1b{ ! 000903000000
081825.391- exit, write len 9
=== End pasted text ===

I don't understand why the select failed, but I don't think I will ever need to insert into this table.

I also don't know if this is a symptom of a larger problem or if this will "fix" my problem (stop the Pro32Srv service from crashing).

I've had this same problem on two glovia pro-iv kernels (4.6 r534 and r523).

[scminter]

This change did not work. Pro32SRV crashed again this morning. I'm turning tracing back on to catch the next crash.

[CSuarezdelReal]

I have not modified any programs.  However, it appears my predecessor changed the function GLOVIA, but never regenned it.  As part of another process GLOVIA had me regen all functions.  So, whatever change he made has now been genned.  I will have to ask Glovia for the original function to compare and see what he did.

May I suggest getting a fresh backup of all your functions before you execute such reggens?.

Assuming the DRIVE file is the fault, I would recommend COPYING the GLOVIA function as it is today, and troubleshoot the COPY (any development or testing environment to do so?). You can then remove the line where the DRIVE file is being processed from the GLOVIA function. I understand GLOVIA (the ERP) would run OK anyway.

When you find what goes wrong, just apply the changes to the live GLOVIA function. That should save you some unnecessary crashes.

Could you post what is the structure of file DRIVE (both, SL and ORACLE)?.

[Vol Yip]

It looks to me the problem not only from the function GLOVIA. It seems that there is a function (or functions) DELETE the file DRIVER and causes GLOVIA to ADD it again. It shouldn't be happened. The systen crashes could be because while a function is deleting (or locking) the file DRIVER while someone logins in and execute GLOVIA, then it tries to ADD the DRIVER back.

I would recommend you do a search to see which functions DELETE (or CHANGE) the file DRIVER.

You should also search all the stored procedures too as GLOVIA uses a lot of stored procedures.

Hope this help.

[scminter]

Thanks for all the help.

I have a new set of trace files from a crash on Friday. It appears the whole DRIVER file issue was a symtpom, not the root problem.

It looks like I have more of a locking issue causing the problem.
Again, nothing in my Oracle alert log or trace logs (??) which is strange since Pro-IV is seeing an ORA-0600.

Trace file snippets:

125039.469-sysfwt: len 14: 1b[ 2 4 ; 1 H 1b[ 0 ; 3 2 m
125039.469-ClientWrite: buflen 14:
125039.469- 1b[ 2 4 ; 1 H 1b[ 0 ; 3 2 m
125039.469-sockSend: sock d7e4, buflen 14:
125039.469- 1b[ 2 4 ; 1 H 1b[ 0 ; 3 2 m
125039.469- exit, write len 14
125039.469-guiHint: 03=UMSG 'WAITING FOR DATABASE RESPONSE - FILE: WO'
125039.469- 1b{ ! 001 030000( W A I T I N G ~ F O R ~ D A T A B A S E ~ R E
125039.469- S P O N S E ~ - ~ F I L E : ~ W O
125039.469-sysCRTWrite : len 49:
125039.469- 1b{ ! 001 030000( W A I T I N G ~ F O R ~ D A T A B A S E ~ R E
125039.469- S P O N S E ~ - ~ F I L E : ~ W O
125039.469-ClientWrite: buflen 49:
125039.469- 1b{ ! 001 030000( W A I T I N G ~ F O R ~ D A T A B A S E ~ R E
125039.469- S P O N S E ~ - ~ F I L E : ~ W O
125039.469-sockSend: sock d7e4, buflen 49:
125039.469- 1b{ ! 001 030000( W A I T I N G ~ F O R ~ D A T A B A S E ~ R E
125039.469- S P O N S E ~ - ~ F I L E : ~ W O
125039.469- exit, write len 49

125039.469-guiHint: 14=FLUSH [reason: Status message]: 1b7b 2100 0714 00
125039.469-sysCRTWrite : len 7: 1b{ ! 00071400
125039.469-ClientWrite: buflen 7:
125039.469- 1b{ ! 00071400
125039.469-sockSend: sock d7e4, buflen 7:
125039.469- 1b{ ! 00071400
125039.469- exit, write len 7
125717.966-OraCmdExecFetch: Execute Failed : XCPT_THROW rc = 600
125717.966-OraSqlError: DBerr 600=>-366: ORA-00600: internal error code, arguments: [], [], [], [], [], [], [], []
125717.966-qgCmdExec: XCPT_RAISE
125717.966-qgSqlFRead: XCPT_CATCH
125717.966- pTib = 12338030, pFcb = 111CC300
125717.966- pTib->pRdCmd = 00000000, pTib->pSndRdCmd = 00000000
125717.966-guiHint: 03=UMSG 'null'
125717.966- 1b{ ! 000903000000
125717.966-sysCRTWrite : len 9: 1b{ ! 000903000000
125717.966-ClientWrite: buflen 9:
125717.966- 1b{ ! 000903000000
125717.966-sockSend: sock d7e4, buflen 9:
125717.966- 1b{ ! 000903000000
125717.966- exit, write len 9


In another trace file:
125552.467- 1b{ ! 005 030000, W A I T I N G ~ F O R ~ D A T A B A S E ~ R E
125552.467- S P O N S E ~ - ~ F I L E : ~ W O _ R T G
125552.467-sysCRTWrite : len 53:
125552.467- 1b{ ! 005 030000, W A I T I N G ~ F O R ~ D A T A B A S E ~ R E
125552.467- S P O N S E ~ - ~ F I L E : ~ W O _ R T G

125552.467-ClientWrite: buflen 53:
125552.467- 1b{ ! 005 030000, W A I T I N G ~ F O R ~ D A T A B A S E ~ R E
125552.467- S P O N S E ~ - ~ F I L E : ~ W O _ R T G
125552.467-sockSend: sock dd60, buflen 53:
125552.467- 1b{ ! 005 030000, W A I T I N G ~ F O R ~ D A T A B A S E ~ R E
125552.467- S P O N S E ~ - ~ F I L E : ~ W O _ R T G
125552.467- exit, write len 53

125552.467-guiHint: 14=FLUSH [reason: Status message]: 1b7b 2100 0714 00
125552.467-sysCRTWrite : len 7: 1b{ ! 00071400
125552.467-ClientWrite: buflen 7:
125552.467- 1b{ ! 00071400
125552.467-sockSend: sock dd60, buflen 7:
125552.467- 1b{ ! 00071400
125552.467- exit, write len 7
125717.966-OraCmdExecFetch: Execute Failed : XCPT_THROW rc = 24364
125717.966-OraSqlError: DBerr 24364=>-366: ORA-24364: internal error while padding blanks

And yet another trace file snippet:
125747.513- 1b{ ! 00: 0300001 P L E A S E ~ W A I T ~ - ~ C O N N E C T I N
125747.513- G ~ T O ~ S Q L ~ D A T A B A S E ~ - ~ O R A C L E
125747.513- exit, write len 58

125747.513-guiHint: 14=FLUSH [reason: Status message]: 1b7b 2100 0714 00
125747.513-sysCRTWrite : len 7: 1b{ ! 00071400
125747.513-ClientWrite: buflen 7:
125747.513- 1b{ ! 00071400
125747.513-sockSend: sock e6e4, buflen 7:
125747.513- 1b{ ! 00071400
125747.513- exit, write len 7
125747.513-OraOpenConn: Enter CriticalSection
125747.529-init_qgdbora_load: Loading DLL 'C:\PRO46\PROORA817.DLL', status 0
125747.529-OraOpenConn: *** Oracle OOCIEnvInit routine Failed
125747.529-OraSqlError: DBerr -360=>-360: OraOpenConn: --> Connect Failed : XCPT_RAISE ***

And another:
125804.404-guiHint: 06=EMSG '360 - SQL ERROR: FATAL: SQL UTILITY ERROR '

[scminter]

Oracle version is 8.1.0.0.

I'm thinking of upgrading to 9i. I believe the problem lies in the interaction between Oracle and the Pro-IV kernel, not in one of the Pro-IV programs. Since I need to upgrade to Oracle 9i anyway I'm thinking of giving it a shot. This may introduce more new problems however.