[Nagiosplug-help] check_nrpe plugins work fine for a while then start throwing "NRPE: unable to read output"

[Kyle Smith]

Hey Philip,

Thanks for the response, see my comments inline...
On May 20, 2009, at 6:09 PM, Philip Marcus wrote:

> Hey Kyle,
>
> I'm just throwing out some random ideas off the top of my head...
> To test, I just enabled debug in one of my nrpe configs, restarted  
> the process and did see more output from the daemon.log file in my / 
> var/log  dir. (running ubuntu)
> Do you see any output from nrpe? you should see a startup and stop  
> message. If not, check the logging facility, and Stop/Start the nrpe  
> daemon again.

Yes I see nrpe messages for startup and shtudown of the daemon in my  
system log file (I'm on OSX so logfile is /var/log/system.log). My  
issue is that I don't see any more messages with debug enabled than I  
do with debug disabled. This is hampering my ability to work this  
issue since i can't tell what nrpe is doing internally. Here's a  
snippet of the logging I see for a shutdown and restart of nrpe on my  
host manta. As part of the startup logging it tells me I have  
configrued nrpe to accept command args which is accurate...

May 21 07:55:30 manta nrpe[10200]: Caught SIGTERM - shutting down...
May 21 07:55:30 manta nrpe[10200]: Cannot remove pidfile '/var/run/ 
nrpe.pid' - check your privileges.
May 21 07:55:30 manta nrpe[10200]: Daemon shutdown
May 21 07:55:33 manta sudo[11969]: kylesmith : TTY=ttys000 ; PWD=/ 
Users/kylesmith ; USER=root ; COMMAND=/usr/local/nagios/etc/init.d/ 
init-script start
May 21 07:55:33 manta nrpe[11971]: Starting up daemon
May 21 07:55:33 manta nrpe[11971]: Warning: Daemon is configured to  
accept command arguments from clients!

>
>
> Regarding the problem where after a while the remote check fails,  
> thats a difficult one to pin down. It sounds like something is  
> timing out. Without knowing anything about your environment or what  
> the actual check is doing..., but it sounds like maybe some local  
> disk access is timing out (I've seen this with kerberized NFS.),  
> maybe some other local environment setting or keyring or something  
> is expiring. When you login (as which ever user), you get a fresh  
> "cert" and the command works, but the cert the existing process is  
> running under has timed out. Like I said, just a stab in the dark.

I understand what you are getting at and I will investigate such  
possibilities.
>
>
> Good luck,

Thanks again! >bows deeply<

kyle

>
>
> Philip
>
>
> On Wed, May 20, 2009 at 3:56 PM, Kyle Smith <kyles at apple.com> wrote:
> Hey Nagios Folks,
>
> We're seeing some strange behavior with a few of our nrpe plugins  
> and am wondering if anyone has any ideas on how to work on the  
> problem.
>
> We are running OS X 10.5.6 with NRPE 2.12 and NRPE plugins 1.4.13.
>
> We have some plugins written in Ruby (boss loves it, I am learning  
> it :-)) that work fine with a freshly started nrpe daemon. Start up  
> nrpe on the client, force checks from the gui, see a green nagios  
> console, nice. But after a while in the console under "Status  
> Information" we start seeing "NRPE: unable to read output" rather  
> than our normal success messages. Bounce the nrpe on the client,  
> force the check through the gui, and things start working again for  
> a while.
>
> While seeing the errors I can still run the plugin on the client no  
> problem but when I try to run it from the server via command line (/ 
> usr/local/nagios/libexec/check_nrpe -H IP.of.client -c command) I  
> see the error "NRPE: unable to read output".
>
> Another data point - We have one system where things were working  
> fine but we needed to start passing params to nrpe. I recompiled  
> after running ./configure --enable-command-args and setting  
> dont_blame_nrpe=1 in the nrpe.cfg. Start up a new nrpe and it works  
> fine, can pass params from the server no problem...but the a check  
> that was working fine for weeks starts throwing "NRPE: unable to  
> read output" after 3 or 4 successful invocations. Arg!! I had to  
> rollback to the previous nrpe and check_nrpe.
>
> I am trying to set nrpe debug on in nrpe.cfg on the client and then  
> bounce nrpe but i don't see any more messages in /var/log/system.log  
> as compared to when I run with nrpe debug turned off. It's like  
> turning on debug has no effect.
>
> I am curious about 2 issues - 1) why no extra messages in the system  
> log file with debug enabled? and 2) Anyone seen anything like this  
> happening where nrpe plugins work fine for a while then start  
> throwing the error?
>
> TIA
>
> kyle
>
>
>
> ------------------------------------------------------------------------------
> Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT
> is a gathering of tech-side developers & brand creativity  
> professionals. Meet
> the minds behind Google Creative Lab, Visual Complexity, Processing, &
> iPhoneDevCamp asthey present alongside digital heavyweights like  
> Barbarian
> Group, R/GA, & Big Spaceship. http://www.creativitycat.com
> _______________________________________________
> Nagiosplug-help mailing list
> Nagiosplug-help at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nagiosplug-help
> ::: Please include plugins version (-v) and OS when reporting any  
> issue.
> ::: Messages without supporting info will risk being sent to /dev/null
>
>
> ------------------------------------------------------------------------------
> Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT
> is a gathering of tech-side developers & brand creativity  
> professionals. Meet
> the minds behind Google Creative Lab, Visual Complexity, Processing, &
> iPhoneDevCamp asthey present alongside digital heavyweights like  
> Barbarian
> Group, R/GA, & Big Spaceship. http://www.creativitycat.com  
> _______________________________________________
> Nagiosplug-help mailing list
> Nagiosplug-help at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nagiosplug-help
> ::: Please include plugins version (-v) and OS when reporting any  
> issue.
> ::: Messages without supporting info will risk being sent to /dev/null

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.monitoring-plugins.org/archive/help/attachments/20090521/9f2abe00/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3980 bytes
Desc: not available
URL: <https://www.monitoring-plugins.org/archive/help/attachments/20090521/9f2abe00/attachment.bin>