3 replies to this topic

[Rick Young]

Please don't anyone take any real time on this - just throwing it out there on the offchance someone has seen this happen and maybe has a good clue or knowledge as to "why". HP support has been contacted, although there has been no feedback/theory thus far.

Client's server has been up for some 320+ days running with no known issues (with one exception, not including the one I'm posting about). The exception was about 2-weeks ago, had a ProIV function coredump a few times filling up root volume.

Last Friday afternoon cronjobs stopped executing. Based on some basic logging from some of the jobs - the prior night overnight jobs all ran; the "hourly type" jobs last ran at 2pm; and an every minute job last ran at 2:02pm. There are a mixture of ProIV transparent login jobs as well as shell scripts; there are no other programs/applications being executed.

User reports first thing Monday morning started the hunt for why "their" expected automatic data updates had not occurred.

- ps -ef |grep cron - showed cron to be running
- /var/adm/syslog/syslog.log - did not show any errors in the Friday ~2pm timeframe, nor the prior 24-hours
- /var/adm/cron/log - simply shows cron jobs running and then no entries.
- dmesg - shows the last "nospace" errors only
I did not check every single job to see if any were actually still running processes - I probably checked about 1/3 of them. But that does of course mean, I cannot guarantee that there was not a "hung" job blocking the queue.
There have been no changes to the crontab, nor the cron job scripts nor the ProIV functions being called in 3 weeks.
The largest interval of any job, is "skip the weekend" - but there are always at least "half a dozen" jobs executed daily and multiple times per day.
Server time had not been changed recently.
So for over 300-days, the jobs have all run fine.

I then used crontab -e to verify that all jobs in the schedule were as expected. Upon quitting, ALL of the jobs then executed - and at ~9:30am on Monday morning last day of the month, the "overnight" kill users process did not rate highly on the popularity list.

Ongoing for the remainder of the day, overnight, and this morning - cronjobs have/are executed/executing normally.

Once again - please don't expend any real time on this - but any thoughts would be appreciated.

Cheers
Rick

[kapoof]

Hi Rick,

I have encoutered problems similar to what you are describing, after a lat of time trying to find the problem it turned out to be the uptime of Unix without a reboot. You can check the uptime without rebout by the command uutime. After rebooting the box no other occurances happened. This also affects ORACLE (ALL versions). Hope this helps.

Best Regards,

Rob.

[Chris Mackenzie]

although you'd expect something in the logs - you don't mention file system status? /var /tmp etc - fsck. Also three weeks is a lot shorter than 320 days. Would be good to know a lot more about that cron process when this is occuring.

You don't mention if this might be a virtual server?

Edited by Chris Mackenzie, 04 February 2011 - 12:36 AM.

[Rick Young]

Hi Rick,

I have encoutered problems similar to what you are describing, after a lat of time trying to find the problem it turned out to be the uptime of Unix without a reboot. You can check the uptime without rebout by the command uutime. After rebooting the box no other occurances happened. This also affects ORACLE (ALL versions). Hope this helps.

Best Regards,

Rob.

although you'd expect something in the logs - you don't mention file system status? /var /tmp etc - fsck. Also three weeks is a lot shorter than 320 days. Would be good to know a lot more about that cron process when this is occuring.

You don't mention if this might be a virtual server?

Thanks for replies guys.

Server is still stable Friday morning. "We" (i.e. client's IT guy, boss and myself) elected to not perform a reboot on the day, due to the server appearing to be stable again and allowing the end-of-month volume of work to continue. There's one scheduled for next Thursday night.

No replies from HP; no other issues noticed nor reported. fsck hasn't been run - but there are no known, other issues with any of the volumes.

As far as I know, last cron change was to bump a copy script to start at 0400 days 0-5 instead of 0430 days 0-5 - this is a copy of production .pro files for "offline" datamining purposes. I am not aware of (i.e. don't remember and cannot find any documentation of..) any other cron changes nor alterations to scripts/programs that are cronjobs. This was a "new" server a year ago, and everything was migrated (whether literally, or manually recreated) from the prior box. This is a physical box rather than a virtual.

I neglected to mention in my op, that the coredumping ProIV function was not a cronjob - and it was recoded.

Cheers
Rick