Tatsuo Indou
tatsu****@mtb*****
2009年 2月 25日 (水) 03:07:21 JST
お世話になります。 VMware上で、CentOS5.2、heartbeatをインストールしました。仮想環境上で、二 代のサーバを作成し、hertbeatを試してみようと考えています。 /etc/ha.d/ha.cfと/etc/ha.d/authkeysを作成し、heartbeatを両系で実行しました。 しばらくの後、hb_monitor を実行してみると、以下のような状態です。 ■CentOS523(正系) ============ Last updated: Wed Feb 25 02:39:50 2009 Current DC: NONE 1 Nodes configured. 0 Resources configured. ============ Node: centos523.tsystem.gr.jp (8be21ed6-2d24-6752-ccad-83ddfe59ae12): online ■CentOS524(副系) ============ Last updated: Wed Feb 25 02:37:58 2009 Current DC: NONE 1 Nodes configured. 0 Resources configured. ============ Node: centos524.tsystem.gr.jp (939d5b4d-bc3b-60ec-e72f-f04d3ef1b5d4): online リソースはまだ定義していません。このときでも、ha.cfに書かれているNodeが 両方ともonlineになると認識しているですが、片側が出力されませんでした。 また、Current DCはどちらも、monitorを実行したホストとなってしまいます。 hb_monitor -e -i 1 -nにより詳細を表示してみると、 いずれもオンライン側には、 === FAIL-COUNT === === INSTANCE-ATTRIBUTES === attribute (probe_complete:true) === INTERCONNECT-LAN === heartbeat-comm (eth1:dead): Heartbeat is failure !! のメッセージが出ています。 ネットワークに問題があると思われ、ifconfigを実行してみますと、eth0(通常 LAN(ブリッジ))、eth1(ハートビート用LAN(ホストオンリー))のいずれも動作し ていると思われます。 互いに、逆側のホストのeth1のIPアドレスにpingは通りますし、sshによる逆側 のホストのeth1のIPアドレスへのログインも可能です。 ha.cf内で、debugを指定したファイル、またログファイルには、 WARN: node 逆のノード: is dead と出ています。逆のノードへの通信がうまくいっていないように思えます。 ha.cf内の、bcast eth1を、ucast eth1 逆側のホストのeth1のIPアドレス と指 定しても同様でした。 初回に動作させたのち、/var/lib/heartbeat/crm/cib.xmlが作成されていました ので、<nodes></nodes>の間に、逆側のnodeをコピーして追記しました。 その時には、crm_mon、hb_monitorともNodeの欄には定義したnodeは出ますが、 通信ができていないようですので、OFFLINEのままです。 Node: centos524.tsystem.gr.jp (939d5b4d-bc3b-60ec-e72f-f04d3ef1b5d4): OFFLINE Node: centos523.tsystem.gr.jp (8be21ed6-2d24-6752-ccad-83ddfe59ae12): OFFLINE eth1経由での通信が正しく動作していないようなのですが、どのようの対応すれ ば良いのでしょうか。 私は、ifconfigでeth1が、UPと表示さること。 逆側のeth1へのpingと、ssh接続で正しくネットワークは動作していると考えて いました。 そのため、対処方法が思いつかない状態です。どなたか、お気づきの点ありまし たら、ご指摘していただけませんか。よろしくお願いします。 ■各ソフトのバージョンです。 ホストOS Windows Vista SP1 VMware VMware Workstation 6.0.5 build-109488 OS CentOS 5.2 x86_64 heartbeat heartbeat-2.1.4-1 heartbeat-gui-2.1.4-1 ■定義ファイルの内容です。 # cat /etc/ha.d/ha.cf # 両系とも同様のファイルです。 crm on debugfile /var/log/ha-debug logfile /var/log/ha-log logfacility local0 keepalive 3 deadtime 30 deadping 40 warntime 10 initdead 60 udpport 694 auto_failback on bcast eth1 node CentOS523 node CentOS524 respawn hacluster /usr/lib64/heartbeat/pingd -m 100 -d 5s -a default_ping_set uuidfrom nodename ping 192.168.2.193 # cat /etc/ha.d/authkeys auth 1 1 sha1 Hello! ■ha-logです。 090225 024431:root @ CentOS523:crm# sed -n "5533,\$p" /var/log/ha-log heartbeat[3853]: 2009/02/25_02:34:12 info: ************************** heartbeat[3853]: 2009/02/25_02:34:12 info: Configuration validated. Starting heartbeat 2.1.4 heartbeat[3854]: 2009/02/25_02:34:12 info: heartbeat: version 2.1.4 heartbeat[3854]: 2009/02/25_02:34:12 info: Heartbeat generation: 1235230397 heartbeat[3854]: 2009/02/25_02:34:12 info: seed is -2044534694 heartbeat[3854]: 2009/02/25_02:34:12 info: glib: UDP Broadcast heartbeat started on port 694 (694) interface eth1 heartbeat[3854]: 2009/02/25_02:34:12 info: glib: UDP Broadcast heartbeat closed on port 694 interface eth1 - Status: 1 heartbeat[3854]: 2009/02/25_02:34:12 info: glib: ping heartbeat started. heartbeat[3854]: 2009/02/25_02:34:12 info: G_main_add_TriggerHandler: Added signal manual handler heartbeat[3854]: 2009/02/25_02:34:12 info: G_main_add_TriggerHandler: Added signal manual handler heartbeat[3854]: 2009/02/25_02:34:12 info: G_main_add_SignalHandler: Added signal handler for signal 17 heartbeat[3854]: 2009/02/25_02:34:12 info: Local status now set to: 'up' heartbeat[3854]: 2009/02/25_02:34:13 info: Link 192.168.2.193:192.168.2.193 up. heartbeat[3854]: 2009/02/25_02:34:13 info: Status update for node 192.168.2.193: status ping heartbeat[3854]: 2009/02/25_02:39:35 WARN: node centos524: is dead heartbeat[3854]: 2009/02/25_02:39:35 info: Comm_now_up(): updating status to active heartbeat[3854]: 2009/02/25_02:39:35 info: Local status now set to: 'active' heartbeat[3854]: 2009/02/25_02:39:35 info: Starting child client "/usr/lib64/heartbeat/ccm" (498,496) heartbeat[3854]: 2009/02/25_02:39:35 info: Starting child client "/usr/lib64/heartbeat/cib" (498,496) heartbeat[3854]: 2009/02/25_02:39:35 info: Starting child client "/usr/lib64/heartbeat/lrmd -r" (0,0) heartbeat[3854]: 2009/02/25_02:39:35 info: Starting child client "/usr/lib64/heartbeat/stonithd" (0,0) heartbeat[3854]: 2009/02/25_02:39:35 info: Starting child client "/usr/lib64/heartbeat/attrd" (498,496) heartbeat[3854]: 2009/02/25_02:39:35 info: Starting child client "/usr/lib64/heartbeat/crmd" (498,496) heartbeat[3854]: 2009/02/25_02:39:35 info: Starting child client "/usr/lib64/heartbeat/mgmtd -v" (0,0) heartbeat[3854]: 2009/02/25_02:39:35 info: Starting child client "/usr/lib64/heartbeat/pingd -m 100 -d 5s -a default_ping_set" (498,496) heartbeat[3914]: 2009/02/25_02:39:35 info: Starting "/usr/lib64/heartbeat/ccm" as uid 498 gid 496 (pid 3914) heartbeat[3915]: 2009/02/25_02:39:35 info: Starting "/usr/lib64/heartbeat/cib" as uid 498 gid 496 (pid 3915) heartbeat[3916]: 2009/02/25_02:39:35 info: Starting "/usr/lib64/heartbeat/lrmd -r" as uid 0 gid 0 (pid 3916) heartbeat[3917]: 2009/02/25_02:39:35 info: Starting "/usr/lib64/heartbeat/stonithd" as uid 0 gid 0 (pid 3917) heartbeat[3918]: 2009/02/25_02:39:35 info: Starting "/usr/lib64/heartbeat/attrd" as uid 498 gid 496 (pid 3918) heartbeat[3919]: 2009/02/25_02:39:36 info: Starting "/usr/lib64/heartbeat/crmd" as uid 498 gid 496 (pid 3919) heartbeat[3921]: 2009/02/25_02:39:36 info: Starting "/usr/lib64/heartbeat/pingd -m 100 -d 5s -a default_ping_set" as uid 498 gid 496 (pid 3921) heartbeat[3920]: 2009/02/25_02:39:36 info: Starting "/usr/lib64/heartbeat/mgmtd -v" as uid 0 gid 0 (pid 3920) cib[3915]: 2009/02/25_02:39:36 info: G_main_add_SignalHandler: Added signal handler for signal 15 lrmd[3916]: 2009/02/25_02:39:36 info: G_main_add_SignalHandler: Added signal handler for signal 15 stonithd[3917]: 2009/02/25_02:39:36 info: G_main_add_SignalHandler: Added signal handler for signal 10 attrd[3918]: 2009/02/25_02:39:36 info: G_main_add_SignalHandler: Added signal handler for signal 15 ccm[3914]: 2009/02/25_02:39:36 info: Hostname: centos523 crmd[3919]: 2009/02/25_02:39:36 info: main: CRM Hg Version: fb84f606a422 tip pingd[3921]: 2009/02/25_02:39:36 info: Invoked: /usr/lib64/heartbeat/pingd -m 100 -d 5s -a default_ping_set mgmtd[3920]: 2009/02/25_02:39:36 info: G_main_add_SignalHandler: Added signal handler for signal 15 cib[3915]: 2009/02/25_02:39:36 info: G_main_add_TriggerHandler: Added signal manual handler lrmd[3916]: 2009/02/25_02:39:36 info: G_main_add_SignalHandler: Added signal handler for signal 17 stonithd[3917]: 2009/02/25_02:39:36 info: G_main_add_SignalHandler: Added signal handler for signal 12 attrd[3918]: 2009/02/25_02:39:36 info: register_with_ha: Hostname: centos523 crmd[3919]: 2009/02/25_02:39:36 info: crmd_init: Starting crmd cib[3915]: 2009/02/25_02:39:36 info: G_main_add_SignalHandler: Added signal handler for signal 17 lrmd[3916]: 2009/02/25_02:39:36 info: G_main_add_SignalHandler: Added signal handler for signal 10 stonithd[3917]: 2009/02/25_02:39:36 info: Signing in with heartbeat. crmd[3919]: 2009/02/25_02:39:36 info: G_main_add_SignalHandler: Added signal handler for signal 15 mgmtd[3920]: 2009/02/25_02:39:36 info: G_main_add_SignalHandler: Added signal handler for signal 10 cib[3915]: 2009/02/25_02:39:36 info: main: Retrieval of a per-action CIB: disabled lrmd[3916]: 2009/02/25_02:39:36 info: G_main_add_SignalHandler: Added signal handler for signal 12 crmd[3919]: 2009/02/25_02:39:36 info: G_main_add_TriggerHandler: Added signal manual handler mgmtd[3920]: 2009/02/25_02:39:36 info: G_main_add_SignalHandler: Added signal handler for signal 12 cib[3915]: 2009/02/25_02:39:36 info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.xml (digest: /var/lib/heartbeat/crm/cib.xml.sig) lrmd[3916]: 2009/02/25_02:39:36 info: Started. crmd[3919]: 2009/02/25_02:39:36 info: G_main_add_SignalHandler: Added signal handler for signal 17 attrd[3918]: 2009/02/25_02:39:36 info: register_with_ha: UUID: 8be21ed6-2d24-6752-ccad-83ddfe59ae12 stonithd[3917]: 2009/02/25_02:39:36 notice: /usr/lib64/heartbeat/stonithd start up successfully. cib[3915]: 2009/02/25_02:39:36 WARN: retrieveCib: Cluster configuration not found: /var/lib/heartbeat/crm/cib.xml mgmtd[3920]: 2009/02/25_02:39:36 info: init_crm stonithd[3917]: 2009/02/25_02:39:36 info: G_main_add_SignalHandler: Added signal handler for signal 17 cib[3915]: 2009/02/25_02:39:36 WARN: readCibXmlFile: Primary configuration corrupt or unusable, trying backup... mgmtd[3920]: 2009/02/25_02:39:36 info: login to cib: 0, ret:-10 cib[3915]: 2009/02/25_02:39:36 info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.xml.last (digest: /var/lib/heartbeat/crm/cib.xml.sig.last) cib[3915]: 2009/02/25_02:39:36 WARN: retrieveCib: Cluster configuration not found: /var/lib/heartbeat/crm/cib.xml.last cib[3915]: 2009/02/25_02:39:36 WARN: readCibXmlFile: Continuing with an empty configuration. cib[3915]: 2009/02/25_02:39:36 WARN: readCibXmlFile: No value for admin_epoch was specified in the configuration. cib[3915]: 2009/02/25_02:39:36 WARN: readCibXmlFile: The reccomended course of action is to shutdown, run crm_verify and fix any errors it reports. cib[3915]: 2009/02/25_02:39:36 WARN: readCibXmlFile: We will default to zero and continue but may get confused about which configuration to use if multiple nodes are powered up at the same time. cib[3915]: 2009/02/25_02:39:36 info: log_data_element: readCibXmlFile: [on-disk] <cib generated="true" admin_epoch="0" epoch="0" num_updates="0" have_quorum="false"> cib[3915]: 2009/02/25_02:39:36 info: log_data_element: readCibXmlFile: [on-disk] <configuration> cib[3915]: 2009/02/25_02:39:36 info: log_data_element: readCibXmlFile: [on-disk] <crm_config/> cib[3915]: 2009/02/25_02:39:36 info: log_data_element: readCibXmlFile: [on-disk] <nodes/> cib[3915]: 2009/02/25_02:39:36 info: log_data_element: readCibXmlFile: [on-disk] <resources/> cib[3915]: 2009/02/25_02:39:36 info: log_data_element: readCibXmlFile: [on-disk] <constraints/> cib[3915]: 2009/02/25_02:39:36 info: log_data_element: readCibXmlFile: [on-disk] </configuration> cib[3915]: 2009/02/25_02:39:36 info: log_data_element: readCibXmlFile: [on-disk] <status/> cib[3915]: 2009/02/25_02:39:36 info: log_data_element: readCibXmlFile: [on-disk] </cib> cib[3915]: 2009/02/25_02:39:36 notice: readCibXmlFile: Enabling DTD validation on the existing (sane) configuration cib[3915]: 2009/02/25_02:39:36 info: startCib: CIB Initialization completed successfully cib[3915]: 2009/02/25_02:39:36 info: cib_register_ha: Signing in with Heartbeat cib[3915]: 2009/02/25_02:39:36 info: cib_register_ha: FSA Hostname: centos523 cib[3915]: 2009/02/25_02:39:36 info: ccm_connect: Registering with CCM... cib[3915]: 2009/02/25_02:39:36 WARN: ccm_connect: CCM Activation failed cib[3915]: 2009/02/25_02:39:36 WARN: ccm_connect: CCM Connection failed 1 times (30 max) cib[3915]: 2009/02/25_02:39:39 info: ccm_connect: Registering with CCM... cib[3915]: 2009/02/25_02:39:39 WARN: ccm_connect: CCM Activation failed cib[3915]: 2009/02/25_02:39:39 WARN: ccm_connect: CCM Connection failed 2 times (30 max) ccm[3914]: 2009/02/25_02:39:40 info: Break tie for 2 nodes cluster ccm[3914]: 2009/02/25_02:39:40 info: G_main_add_SignalHandler: Added signal handler for signal 15 cib[3915]: 2009/02/25_02:39:42 info: ccm_connect: Registering with CCM... cib[3915]: 2009/02/25_02:39:43 info: cib_init: Starting cib mainloop cib[3915]: 2009/02/25_02:39:43 info: mem_handle_event: Got an event OC_EV_MS_NEW_MEMBERSHIP from ccm cib[3915]: 2009/02/25_02:39:43 info: mem_handle_event: instance=1, nodes=1, new=1, lost=0, n_idx=0, new_idx=0, old_idx=3 cib[3915]: 2009/02/25_02:39:43 info: cib_ccm_msg_callback: PEER: centos523 crmd[3919]: 2009/02/25_02:39:43 info: do_cib_control: CIB connection established crmd[3919]: 2009/02/25_02:39:43 info: register_with_ha: Hostname: centos523 crmd[3919]: 2009/02/25_02:39:43 info: register_with_ha: UUID: 8be21ed6-2d24-6752-ccad-83ddfe59ae12 mgmtd[3920]: 2009/02/25_02:39:43 info: Started. cib[3915]: 2009/02/25_02:39:43 info: cib_null_callback: Setting cib_refresh_notify callbacks for crmd: on cib[3915]: 2009/02/25_02:39:43 info: cib_null_callback: Setting cib_diff_notify callbacks for mgmtd: on cib[3915]: 2009/02/25_02:39:43 info: cib_client_status_callback: Status update: Client centos523/cib now has status [join] cib[3915]: 2009/02/25_02:39:43 info: cib_client_status_callback: Status update: Client centos523/cib now has status [online] cib[3922]: 2009/02/25_02:39:43 info: write_cib_contents: Wrote version 0.0.0 of the CIB to disk (digest: d79c45011201386a3384cc4233eb082a) cib[3922]: 2009/02/25_02:39:43 info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.xml (digest: /var/lib/heartbeat/crm/cib.xml.sig) crmd[3919]: 2009/02/25_02:39:43 info: populate_cib_nodes: Requesting the list of configured nodes crmd[3919]: 2009/02/25_02:39:43 WARN: get_uuid: Could not calculate UUID for centos524 crmd[3919]: 2009/02/25_02:39:43 WARN: populate_cib_nodes: Node centos524: no uuid found crmd[3919]: 2009/02/25_02:39:44 notice: populate_cib_nodes: Node: centos523 (uuid: 8be21ed6-2d24-6752-ccad-83ddfe59ae12) crmd[3919]: 2009/02/25_02:39:44 info: do_ha_control: Connected to Heartbeat crmd[3919]: 2009/02/25_02:39:44 info: do_ccm_control: CCM connection established... waiting for first callback crmd[3919]: 2009/02/25_02:39:44 info: do_started: Delaying start, CCM (0000000000100000) not connected crmd[3919]: 2009/02/25_02:39:44 info: crmd_init: Starting crmd's mainloop crmd[3919]: 2009/02/25_02:39:44 notice: crmd_client_status_callback: Status update: Client centos523/crmd now has status [online] cib[3923]: 2009/02/25_02:39:44 info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.xml (digest: /var/lib/heartbeat/crm/cib.xml.sig) cib[3923]: 2009/02/25_02:39:44 info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.xml (digest: /var/lib/heartbeat/crm/cib.xml.sig) cib[3923]: 2009/02/25_02:39:44 info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.xml.last (digest: /var/lib/heartbeat/crm/cib.xml.sig.last) cib[3923]: 2009/02/25_02:39:44 info: write_cib_contents: Wrote version 0.0.0 of the CIB to disk (digest: 004c5aa8495ded4f46bf66fa4f2a6825) cib[3923]: 2009/02/25_02:39:44 info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.xml (digest: /var/lib/heartbeat/crm/cib.xml.sig) cib[3923]: 2009/02/25_02:39:44 info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.xml.last (digest: /var/lib/heartbeat/crm/cib.xml.sig.last) crmd[3919]: 2009/02/25_02:39:44 notice: crmd_client_status_callback: Status update: Client centos523/crmd now has status [online] crmd[3919]: 2009/02/25_02:39:45 info: do_started: Delaying start, CCM (0000000000100000) not connected crmd[3919]: 2009/02/25_02:39:45 info: mem_handle_event: Got an event OC_EV_MS_NEW_MEMBERSHIP from ccm crmd[3919]: 2009/02/25_02:39:45 info: mem_handle_event: instance=1, nodes=1, new=1, lost=0, n_idx=0, new_idx=0, old_idx=3 crmd[3919]: 2009/02/25_02:39:45 info: crmd_ccm_msg_callback: Quorum (re)attained after event=NEW MEMBERSHIP (id=1) crmd[3919]: 2009/02/25_02:39:45 info: ccm_event_detail: NEW MEMBERSHIP: trans=1, nodes=1, new=1, lost=0 n_idx=0, new_idx=0, old_idx=3 crmd[3919]: 2009/02/25_02:39:45 info: ccm_event_detail: CURRENT: centos523 [nodeid=0, born=1] crmd[3919]: 2009/02/25_02:39:45 info: ccm_event_detail: NEW: centos523 [nodeid=0, born=1] crmd[3919]: 2009/02/25_02:39:45 info: do_started: The local CRM is operational crmd[3919]: 2009/02/25_02:39:45 info: do_state_transition: State transition S_STARTING -> S_PENDING [ input=I_PENDING cause=C_CCM_CALLBACK origin=do_started ] attrd[3918]: 2009/02/25_02:39:48 info: main: Starting mainloop... heartbeat[3854]: 2009/02/25_02:39:51 WARN: Client [pingd] pid 3921 failed authorization [client failed authorization] heartbeat[3854]: 2009/02/25_02:39:51 ERROR: api_process_registration_msg: cannot add client(pingd) pingd[3921]: 2009/02/25_02:39:51 ERROR: register_with_ha: Cannot sign on with heartbeat: pingd[3921]: 2009/02/25_02:39:51 ERROR: register_with_ha: REASON: pingd[3921]: 2009/02/25_02:39:51 ERROR: main: HA registration failed heartbeat[3854]: 2009/02/25_02:39:51 WARN: Managed /usr/lib64/heartbeat/pingd -m 100 -d 5s -a default_ping_set process 3921 exited with return code 1. heartbeat[3854]: 2009/02/25_02:39:51 ERROR: Respawning client "/usr/lib64/heartbeat/pingd -m 100 -d 5s -a default_ping_set": heartbeat[3854]: 2009/02/25_02:39:51 info: Starting child client "/usr/lib64/heartbeat/pingd -m 100 -d 5s -a default_ping_set" (498,496) heartbeat[3924]: 2009/02/25_02:39:52 info: Starting "/usr/lib64/heartbeat/pingd -m 100 -d 5s -a default_ping_set" as uid 498 gid 496 (pid 3924) pingd[3924]: 2009/02/25_02:39:52 info: Invoked: /usr/lib64/heartbeat/pingd -m 100 -d 5s -a default_ping_set heartbeat[3854]: 2009/02/25_02:39:57 WARN: Client [pingd] pid 3924 failed authorization [client failed authorization] heartbeat[3854]: 2009/02/25_02:39:57 ERROR: api_process_registration_msg: cannot add client(pingd) pingd[3924]: 2009/02/25_02:39:57 ERROR: register_with_ha: Cannot sign on with heartbeat: pingd[3924]: 2009/02/25_02:39:57 ERROR: register_with_ha: REASON: pingd[3924]: 2009/02/25_02:39:57 ERROR: main: HA registration failed heartbeat[3854]: 2009/02/25_02:39:57 WARN: Managed /usr/lib64/heartbeat/pingd -m 100 -d 5s -a default_ping_set process 3924 exited with return code 1. heartbeat[3854]: 2009/02/25_02:39:57 ERROR: Respawning client "/usr/lib64/heartbeat/pingd -m 100 -d 5s -a default_ping_set": heartbeat[3854]: 2009/02/25_02:39:57 info: Starting child client "/usr/lib64/heartbeat/pingd -m 100 -d 5s -a default_ping_set" (498,496) heartbeat[3925]: 2009/02/25_02:39:58 info: Starting "/usr/lib64/heartbeat/pingd -m 100 -d 5s -a default_ping_set" as uid 498 gid 496 (pid 3925) pingd[3925]: 2009/02/25_02:39:58 info: Invoked: /usr/lib64/heartbeat/pingd -m 100 -d 5s -a default_ping_set heartbeat[3854]: 2009/02/25_02:41:02 WARN: Client [pingd] pid 3925 failed authorization [client failed authorization] heartbeat[3854]: 2009/02/25_02:41:02 ERROR: api_process_registration_msg: cannot add client(pingd) pingd[3925]: 2009/02/25_02:41:02 ERROR: register_with_ha: Cannot sign on with heartbeat: pingd[3925]: 2009/02/25_02:41:02 ERROR: register_with_ha: REASON: pingd[3925]: 2009/02/25_02:41:02 ERROR: main: HA registration failed heartbeat[3854]: 2009/02/25_02:41:02 WARN: Managed /usr/lib64/heartbeat/pingd -m 100 -d 5s -a default_ping_set process 3925 exited with return code 1. heartbeat[3854]: 2009/02/25_02:41:02 ERROR: Respawning client "/usr/lib64/heartbeat/pingd -m 100 -d 5s -a default_ping_set": heartbeat[3854]: 2009/02/25_02:41:02 info: Starting child client "/usr/lib64/heartbeat/pingd -m 100 -d 5s -a default_ping_set" (498,496) heartbeat[3948]: 2009/02/25_02:41:03 info: Starting "/usr/lib64/heartbeat/pingd -m 100 -d 5s -a default_ping_set" as uid 498 gid 496 (pid 3948) pingd[3948]: 2009/02/25_02:41:03 info: Invoked: /usr/lib64/heartbeat/pingd -m 100 -d 5s -a default_ping_set heartbeat[3854]: 2009/02/25_02:41:08 WARN: Client [pingd] pid 3948 failed authorization [client failed authorization] heartbeat[3854]: 2009/02/25_02:41:08 ERROR: api_process_registration_msg: cannot add client(pingd) pingd[3948]: 2009/02/25_02:41:08 ERROR: register_with_ha: Cannot sign on with heartbeat: pingd[3948]: 2009/02/25_02:41:08 ERROR: register_with_ha: REASON: pingd[3948]: 2009/02/25_02:41:08 ERROR: main: HA registration failed heartbeat[3854]: 2009/02/25_02:41:08 WARN: Managed /usr/lib64/heartbeat/pingd -m 100 -d 5s -a default_ping_set process 3948 exited with return code 1. heartbeat[3854]: 2009/02/25_02:41:08 ERROR: Respawning client "/usr/lib64/heartbeat/pingd -m 100 -d 5s -a default_ping_set": heartbeat[3854]: 2009/02/25_02:41:08 info: Starting child client "/usr/lib64/heartbeat/pingd -m 100 -d 5s -a default_ping_set" (498,496) heartbeat[3949]: 2009/02/25_02:41:09 info: Starting "/usr/lib64/heartbeat/pingd -m 100 -d 5s -a default_ping_set" as uid 498 gid 496 (pid 3949) pingd[3949]: 2009/02/25_02:41:09 info: Invoked: /usr/lib64/heartbeat/pingd -m 100 -d 5s -a default_ping_set heartbeat[3854]: 2009/02/25_02:41:14 WARN: Client [pingd] pid 3949 failed authorization [client failed authorization] heartbeat[3854]: 2009/02/25_02:41:14 ERROR: api_process_registration_msg: cannot add client(pingd) pingd[3949]: 2009/02/25_02:41:14 ERROR: register_with_ha: Cannot sign on with heartbeat: pingd[3949]: 2009/02/25_02:41:14 ERROR: register_with_ha: REASON: pingd[3949]: 2009/02/25_02:41:14 ERROR: main: HA registration failed heartbeat[3854]: 2009/02/25_02:41:14 WARN: Managed /usr/lib64/heartbeat/pingd -m 100 -d 5s -a default_ping_set process 3949 exited with return code 1. heartbeat[3854]: 2009/02/25_02:41:14 ERROR: Respawning client "/usr/lib64/heartbeat/pingd -m 100 -d 5s -a default_ping_set": heartbeat[3854]: 2009/02/25_02:41:14 info: Starting child client "/usr/lib64/heartbeat/pingd -m 100 -d 5s -a default_ping_set" (498,496) heartbeat[3950]: 2009/02/25_02:41:15 info: Starting "/usr/lib64/heartbeat/pingd -m 100 -d 5s -a default_ping_set" as uid 498 gid 496 (pid 3950) pingd[3950]: 2009/02/25_02:41:15 info: Invoked: /usr/lib64/heartbeat/pingd -m 100 -d 5s -a default_ping_set heartbeat[3854]: 2009/02/25_02:41:20 WARN: Client [pingd] pid 3950 failed authorization [client failed authorization] heartbeat[3854]: 2009/02/25_02:41:20 ERROR: api_process_registration_msg: cannot add client(pingd) pingd[3950]: 2009/02/25_02:41:20 ERROR: register_with_ha: Cannot sign on with heartbeat: pingd[3950]: 2009/02/25_02:41:20 ERROR: register_with_ha: REASON: pingd[3950]: 2009/02/25_02:41:20 ERROR: main: HA registration failed heartbeat[3854]: 2009/02/25_02:41:20 WARN: Managed /usr/lib64/heartbeat/pingd -m 100 -d 5s -a default_ping_set process 3950 exited with return code 1. heartbeat[3854]: 2009/02/25_02:41:20 ERROR: Respawning client "/usr/lib64/heartbeat/pingd -m 100 -d 5s -a default_ping_set": heartbeat[3854]: 2009/02/25_02:41:20 info: Starting child client "/usr/lib64/heartbeat/pingd -m 100 -d 5s -a default_ping_set" (498,496) heartbeat[3951]: 2009/02/25_02:41:21 info: Starting "/usr/lib64/heartbeat/pingd -m 100 -d 5s -a default_ping_set" as uid 498 gid 496 (pid 3951) pingd[3951]: 2009/02/25_02:41:21 info: Invoked: /usr/lib64/heartbeat/pingd -m 100 -d 5s -a default_ping_set heartbeat[3854]: 2009/02/25_02:41:26 WARN: Client [pingd] pid 3951 failed authorization [client failed authorization] heartbeat[3854]: 2009/02/25_02:41:26 ERROR: api_process_registration_msg: cannot add client(pingd) pingd[3951]: 2009/02/25_02:41:26 ERROR: register_with_ha: Cannot sign on with heartbeat: pingd[3951]: 2009/02/25_02:41:26 ERROR: register_with_ha: REASON: pingd[3951]: 2009/02/25_02:41:26 ERROR: main: HA registration failed heartbeat[3854]: 2009/02/25_02:41:26 WARN: Managed /usr/lib64/heartbeat/pingd -m 100 -d 5s -a default_ping_set process 3951 exited with return code 1. heartbeat[3854]: 2009/02/25_02:41:26 ERROR: Respawning client "/usr/lib64/heartbeat/pingd -m 100 -d 5s -a default_ping_set": heartbeat[3854]: 2009/02/25_02:41:26 info: Starting child client "/usr/lib64/heartbeat/pingd -m 100 -d 5s -a default_ping_set" (498,496) heartbeat[3960]: 2009/02/25_02:41:27 info: Starting "/usr/lib64/heartbeat/pingd -m 100 -d 5s -a default_ping_set" as uid 498 gid 496 (pid 3960) pingd[3960]: 2009/02/25_02:41:27 info: Invoked: /usr/lib64/heartbeat/pingd -m 100 -d 5s -a default_ping_set heartbeat[3854]: 2009/02/25_02:41:32 WARN: Client [pingd] pid 3960 failed authorization [client failed authorization] heartbeat[3854]: 2009/02/25_02:41:32 ERROR: api_process_registration_msg: cannot add client(pingd) pingd[3960]: 2009/02/25_02:41:32 ERROR: register_with_ha: Cannot sign on with heartbeat: pingd[3960]: 2009/02/25_02:41:32 ERROR: register_with_ha: REASON: pingd[3960]: 2009/02/25_02:41:32 ERROR: main: HA registration failed heartbeat[3854]: 2009/02/25_02:41:32 WARN: Managed /usr/lib64/heartbeat/pingd -m 100 -d 5s -a default_ping_set process 3960 exited with return code 1. heartbeat[3854]: 2009/02/25_02:41:32 ERROR: Respawning client "/usr/lib64/heartbeat/pingd -m 100 -d 5s -a default_ping_set": heartbeat[3854]: 2009/02/25_02:41:32 info: Starting child client "/usr/lib64/heartbeat/pingd -m 100 -d 5s -a default_ping_set" (498,496) heartbeat[3979]: 2009/02/25_02:41:33 info: Starting "/usr/lib64/heartbeat/pingd -m 100 -d 5s -a default_ping_set" as uid 498 gid 496 (pid 3979) pingd[3979]: 2009/02/25_02:41:33 info: Invoked: /usr/lib64/heartbeat/pingd -m 100 -d 5s -a default_ping_set heartbeat[3854]: 2009/02/25_02:41:38 WARN: Client [pingd] pid 3979 failed authorization [client failed authorization] heartbeat[3854]: 2009/02/25_02:41:38 ERROR: api_process_registration_msg: cannot add client(pingd) pingd[3979]: 2009/02/25_02:41:38 ERROR: register_with_ha: Cannot sign on with heartbeat: pingd[3979]: 2009/02/25_02:41:38 ERROR: register_with_ha: REASON: pingd[3979]: 2009/02/25_02:41:38 ERROR: main: HA registration failed heartbeat[3854]: 2009/02/25_02:41:38 WARN: Managed /usr/lib64/heartbeat/pingd -m 100 -d 5s -a default_ping_set process 3979 exited with return code 1. heartbeat[3854]: 2009/02/25_02:41:38 ERROR: Respawning client "/usr/lib64/heartbeat/pingd -m 100 -d 5s -a default_ping_set": heartbeat[3854]: 2009/02/25_02:41:38 info: Starting child client "/usr/lib64/heartbeat/pingd -m 100 -d 5s -a default_ping_set" (498,496) heartbeat[3980]: 2009/02/25_02:41:39 info: Starting "/usr/lib64/heartbeat/pingd -m 100 -d 5s -a default_ping_set" as uid 498 gid 496 (pid 3980) pingd[3980]: 2009/02/25_02:41:39 info: Invoked: /usr/lib64/heartbeat/pingd -m 100 -d 5s -a default_ping_set heartbeat[3854]: 2009/02/25_02:41:44 WARN: Client [pingd] pid 3980 failed authorization [client failed authorization] heartbeat[3854]: 2009/02/25_02:41:44 ERROR: api_process_registration_msg: cannot add client(pingd) pingd[3980]: 2009/02/25_02:41:44 ERROR: register_with_ha: Cannot sign on with heartbeat: pingd[3980]: 2009/02/25_02:41:44 ERROR: register_with_ha: REASON: pingd[3980]: 2009/02/25_02:41:44 ERROR: main: HA registration failed heartbeat[3854]: 2009/02/25_02:41:44 WARN: Managed /usr/lib64/heartbeat/pingd -m 100 -d 5s -a default_ping_set process 3980 exited with return code 1. heartbeat[3854]: 2009/02/25_02:41:44 ERROR: Respawning client "/usr/lib64/heartbeat/pingd -m 100 -d 5s -a default_ping_set": heartbeat[3854]: 2009/02/25_02:41:44 info: Starting child client "/usr/lib64/heartbeat/pingd -m 100 -d 5s -a default_ping_set" (498,496) crmd[3919]: 2009/02/25_02:41:45 info: crm_timer_popped: Election Trigger (I_DC_TIMEOUT) just popped! crmd[3919]: 2009/02/25_02:41:45 WARN: do_log: [[FSA]] Input I_DC_TIMEOUT from crm_timer_popped() received in state (S_PENDING) crmd[3919]: 2009/02/25_02:41:45 info: do_state_transition: State transition S_PENDING -> S_ELECTION [ input=I_DC_TIMEOUT cause=C_TIMER_POPPED origin=crm_timer_popped ] crmd[3919]: 2009/02/25_02:41:45 info: do_election_count_vote: Updated voted hash for centos523 to vote crmd[3919]: 2009/02/25_02:41:45 info: do_election_count_vote: Election ignore: our vote (centos523) crmd[3919]: 2009/02/25_02:41:45 info: do_state_transition: State transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC cause=C_FSA_INTERNAL origin=do_election_check ] crmd[3919]: 2009/02/25_02:41:45 info: start_subsystem: Starting sub-system "tengine" crmd[3919]: 2009/02/25_02:41:45 info: start_subsystem: Starting sub-system "pengine" crmd[3919]: 2009/02/25_02:41:45 info: do_dc_takeover: Taking over DC status for this partition tengine[3982]: 2009/02/25_02:41:45 info: G_main_add_SignalHandler: Added signal handler for signal 15 tengine[3982]: 2009/02/25_02:41:45 info: G_main_add_TriggerHandler: Added signal manual handler tengine[3982]: 2009/02/25_02:41:45 info: G_main_add_TriggerHandler: Added signal manual handler cib[3915]: 2009/02/25_02:41:45 info: cib_process_readwrite: We are now in R/W mode pengine[3983]: 2009/02/25_02:41:45 info: G_main_add_SignalHandler: Added signal handler for signal 15 cib[3915]: 2009/02/25_02:41:45 info: revision_check: Updating CIB revision to 2.0 cib[3915]: 2009/02/25_02:41:45 info: log_data_element: cib:diff: - <cib epoch="0" num_updates="0"/> cib[3915]: 2009/02/25_02:41:45 info: log_data_element: cib:diff: + <cib epoch="1" num_updates="1" crm_feature_set="2.0"/> pengine[3983]: 2009/02/25_02:41:45 info: pe_init: Starting pengine cib[3984]: 2009/02/25_02:41:45 info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.xml (digest: /var/lib/heartbeat/crm/cib.xml.sig) cib[3915]: 2009/02/25_02:41:45 info: log_data_element: cib:diff: - <cib epoch="1"/> cib[3984]: 2009/02/25_02:41:45 info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.xml (digest: /var/lib/heartbeat/crm/cib.xml.sig) cib[3915]: 2009/02/25_02:41:45 info: log_data_element: cib:diff: + <cib epoch="2"> crmd[3919]: 2009/02/25_02:41:45 info: erase_node_from_join: Removed dead node centos523 from join calculations: welcomed=0 itegrated=0 finalized=0 confirmed=0 crmd[3919]: 2009/02/25_02:41:45 info: join_make_offer: Making join offers based on membership 1 cib[3915]: 2009/02/25_02:41:45 info: log_data_element: cib:diff: + <configuration> crmd[3919]: 2009/02/25_02:41:45 info: do_dc_join_offer_all: join-1: Waiting on 1 outstanding join acks cib[3984]: 2009/02/25_02:41:45 info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.xml.last (digest: /var/lib/heartbeat/crm/cib.xml.sig.last) cib[3915]: 2009/02/25_02:41:45 info: log_data_element: cib:diff: + <crm_config> cib[3915]: 2009/02/25_02:41:45 info: log_data_element: cib:diff: + <cluster_property_set id="cib-bootstrap-options"> cib[3984]: 2009/02/25_02:41:45 info: write_cib_contents: Wrote version 0.1.1 of the CIB to disk (digest: 7623f0d22eac32f8f7755d7488451f0d) cib[3915]: 2009/02/25_02:41:45 info: log_data_element: cib:diff: + <attributes> cib[3915]: 2009/02/25_02:41:45 info: log_data_element: cib:diff: + <nvpair id="cib-bootstrap-options-dc-version" name="dc-version" value="2.1.4-fb84f606a422 tip"/> cib[3984]: 2009/02/25_02:41:45 info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.xml (digest: /var/lib/heartbeat/crm/cib.xml.sig) cib[3915]: 2009/02/25_02:41:45 info: log_data_element: cib:diff: + </attributes> cib[3984]: 2009/02/25_02:41:45 info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.xml.last (digest: /var/lib/heartbeat/crm/cib.xml.sig.last) cib[3915]: 2009/02/25_02:41:45 info: log_data_element: cib:diff: + </cluster_property_set> cib[3915]: 2009/02/25_02:41:45 info: log_data_element: cib:diff: + </crm_config> cib[3915]: 2009/02/25_02:41:45 info: log_data_element: cib:diff: + </configuration> cib[3915]: 2009/02/25_02:41:45 info: log_data_element: cib:diff: + </cib> cib[3985]: 2009/02/25_02:41:45 info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.xml (digest: /var/lib/heartbeat/crm/cib.xml.sig) tengine[3982]: 2009/02/25_02:41:45 info: te_init: Registering TE UUID: dc33fead-fef6-4cf3-9c45-4b8f8b030d7a cib[3915]: 2009/02/25_02:41:45 info: cib_null_callback: Setting cib_diff_notify callbacks for tengine: on cib[3985]: 2009/02/25_02:41:45 info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.xml (digest: /var/lib/heartbeat/crm/cib.xml.sig) tengine[3982]: 2009/02/25_02:41:45 info: set_graph_functions: Setting custom graph functions cib[3985]: 2009/02/25_02:41:45 info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.xml.last (digest: /var/lib/heartbeat/crm/cib.xml.sig.last) tengine[3982]: 2009/02/25_02:41:45 info: unpack_graph: Unpacked transition -1: 0 actions in 0 synapses tengine[3982]: 2009/02/25_02:41:45 info: te_init: Starting tengine cib[3985]: 2009/02/25_02:41:45 info: write_cib_contents: Wrote version 0.2.1 of the CIB to disk (digest: 03009f6eeefa3cc1ab09948bd50c784a) tengine[3982]: 2009/02/25_02:41:45 info: te_connect_stonith: Attempting connection to fencing daemon... cib[3985]: 2009/02/25_02:41:45 info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.xml (digest: /var/lib/heartbeat/crm/cib.xml.sig) cib[3985]: 2009/02/25_02:41:45 info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.xml.last (digest: /var/lib/heartbeat/crm/cib.xml.sig.last) crmd[3919]: 2009/02/25_02:41:45 info: update_dc: Set DC to centos523 (2.0) crmd[3919]: 2009/02/25_02:41:45 info: do_state_transition: State transition S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state ] crmd[3919]: 2009/02/25_02:41:45 info: do_state_transition: All 1 cluster nodes responded to the join offer. crmd[3919]: 2009/02/25_02:41:45 info: update_attrd: Connecting to attrd... cib[3915]: 2009/02/25_02:41:45 info: sync_our_cib: Syncing CIB to all peers cib[3915]: 2009/02/25_02:41:45 info: log_data_element: cib:diff: - <cib epoch="2"/> cib[3915]: 2009/02/25_02:41:45 info: log_data_element: cib:diff: + <cib epoch="3" dc_uuid="8be21ed6-2d24-6752-ccad-83ddfe59ae12"/> attrd[3918]: 2009/02/25_02:41:45 info: attrd_local_callback: Sending full refresh cib[3986]: 2009/02/25_02:41:45 info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.xml (digest: /var/lib/heartbeat/crm/cib.xml.sig) cib[3986]: 2009/02/25_02:41:45 info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.xml (digest: /var/lib/heartbeat/crm/cib.xml.sig) cib[3986]: 2009/02/25_02:41:45 info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.xml.last (digest: /var/lib/heartbeat/crm/cib.xml.sig.last) cib[3986]: 2009/02/25_02:41:45 info: write_cib_contents: Wrote version 0.3.2 of the CIB to disk (digest: 1539a50eabbfdf328424b2b6197e3ad4) cib[3986]: 2009/02/25_02:41:45 info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.xml (digest: /var/lib/heartbeat/crm/cib.xml.sig) cib[3986]: 2009/02/25_02:41:45 info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.xml.last (digest: /var/lib/heartbeat/crm/cib.xml.sig.last) crmd[3919]: 2009/02/25_02:41:45 info: update_dc: Set DC to centos523 (2.0) heartbeat[3981]: 2009/02/25_02:41:45 info: Starting "/usr/lib64/heartbeat/pingd -m 100 -d 5s -a default_ping_set" as uid 498 gid 496 (pid 3981) pingd[3981]: 2009/02/25_02:41:45 info: Invoked: /usr/lib64/heartbeat/pingd -m 100 -d 5s -a default_ping_set crmd[3919]: 2009/02/25_02:41:46 info: do_dc_join_ack: join-1: Updating node state to member for centos523 crmd[3919]: 2009/02/25_02:41:46 info: do_state_transition: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE [ input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state ] crmd[3919]: 2009/02/25_02:41:46 info: do_state_transition: All 1 cluster nodes are eligible to run resources. tengine[3982]: 2009/02/25_02:41:46 info: te_connect_stonith: Connected tengine[3982]: 2009/02/25_02:41:46 info: update_abort_priority: Abort priority upgraded to 1000000 tengine[3982]: 2009/02/25_02:41:46 info: update_abort_priority: 'DC Takeover' abort superceeded pengine[3983]: 2009/02/25_02:41:46 info: determine_online_status: Node centos523 is online crmd[3919]: 2009/02/25_02:41:46 info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=route_message ] tengine[3982]: 2009/02/25_02:41:46 info: process_te_message: Processing graph derived from /var/lib/heartbeat/pengine/pe-input-65.bz2 tengine[3982]: 2009/02/25_02:41:46 info: unpack_graph: Unpacked transition 0: 1 actions in 1 synapses tengine[3982]: 2009/02/25_02:41:46 info: send_rsc_command: Initiating action 2: probe_complete probe_complete on centos523 tengine[3982]: 2009/02/25_02:41:46 info: run_graph: Transition 0: (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=0) tengine[3982]: 2009/02/25_02:41:46 info: notify_crmd: Transition 0 status: te_complete - <null> crmd[3919]: 2009/02/25_02:41:46 info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_IPC_MESSAGE origin=route_message ] tengine[3982]: 2009/02/25_02:41:46 info: extract_event: Aborting on transient_attributes changes for 8be21ed6-2d24-6752-ccad-83ddfe59ae12 tengine[3982]: 2009/02/25_02:41:46 info: update_abort_priority: Abort priority upgraded to 1000000 crmd[3919]: 2009/02/25_02:41:46 info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_IPC_MESSAGE origin=route_message ] pengine[3983]: 2009/02/25_02:41:46 info: process_pe_message: Transition 0: PEngine Input stored in: /var/lib/heartbeat/pengine/pe-input-65.bz2 crmd[3919]: 2009/02/25_02:41:46 info: do_state_transition: All 1 cluster nodes are eligible to run resources. pengine[3983]: 2009/02/25_02:41:46 info: determine_online_status: Node centos523 is online crmd[3919]: 2009/02/25_02:41:46 info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=route_message ] pengine[3983]: 2009/02/25_02:41:46 info: process_pe_message: Transition 1: PEngine Input stored in: /var/lib/heartbeat/pengine/pe-input-66.bz2 tengine[3982]: 2009/02/25_02:41:46 info: process_te_message: Processing graph derived from /var/lib/heartbeat/pengine/pe-input-66.bz2 tengine[3982]: 2009/02/25_02:41:46 info: unpack_graph: Unpacked transition 1: 0 actions in 0 synapses tengine[3982]: 2009/02/25_02:41:46 info: run_graph: Transition 1: (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0) tengine[3982]: 2009/02/25_02:41:46 info: notify_crmd: Transition 1 status: te_complete - <null> crmd[3919]: 2009/02/25_02:41:46 info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_IPC_MESSAGE origin=route_message ] heartbeat[3854]: 2009/02/25_02:41:50 WARN: Client [pingd] pid 3981 failed authorization [client failed authorization] heartbeat[3854]: 2009/02/25_02:41:50 ERROR: api_process_registration_msg: cannot add client(pingd) pingd[3981]: 2009/02/25_02:41:50 ERROR: register_with_ha: Cannot sign on with heartbeat: pingd[3981]: 2009/02/25_02:41:50 ERROR: register_with_ha: REASON: pingd[3981]: 2009/02/25_02:41:50 ERROR: main: HA registration failed heartbeat[3854]: 2009/02/25_02:41:50 WARN: Managed /usr/lib64/heartbeat/pingd -m 100 -d 5s -a default_ping_set process 3981 exited with return code 1. heartbeat[3854]: 2009/02/25_02:41:50 ERROR: Client /usr/lib64/heartbeat/pingd -m 100 -d 5s -a default_ping_set "respawning too fast" 090225 024440:root @ CentOS523:crm#