New Install Issue - App Tier not reachable

[hawaii_desperado]

I am working on a clean install on a three machine architecture.
My Application tier is failing when I try to ping the distributer.

The Content Manager Machine is Installed, configed and working fine.
The Application Tier is Installed, Config starts, test succeed but I can't ping the distributer with http://appurl:9300/p2pd/servlet, the content manager pings just fine.

This install is on a three machine architecture.
Gateway : RHEL5 32bit - Apache Webserver
App Tier : RHEL 64bit
CM : RHEL 64bit - Oracle 11g CMDB

Any Help would be appreciated.

Mahalo,
Ron

The log is full of these messages
cogserver.log:172.16.4.103:9300 20884   2011-07-15 12:57:45.573 -10     na      na                      pool-1-thread-1 DISP4707     1       Audit.Other.dispatcher.DISP.pogo        pogo    com.cognos.pogo.monitoring.jmx.threshold.ConsolidatedHealthMB
eanCollection           Failure <messages><message><messageString>DPR-DPR-1035 Dispatcher detected an error.</messageString><
/message><message><messageString>null</messageString></message></messages>      Exception creating proxy health MBean for com
.cognos:type=Metrics,group=System,service=reportServicenulljava.lang.SecurityException  at com.cognos.pogo.monitoring.jmx.Pog
oJMXAuthenticator.authenticate(PogoJMXAuthenticator.java:56)    at mx4j.tools.remote.AbstractConnectionManager.authenticate(A
bstractConnectionManager.java:227)      at mx4j.tools.remote.AbstractConnectionManager.connect(AbstractConnectionManager.java
:63)    at mx4j.tools.remote.http.HTTPService.connect(HTTPService.java:60)      at sun.reflect.GeneratedMethodAccessor41.invo
ke(Unknown Source)      at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)      at java.lang.reflect.Method.i
nvoke(Unknown Source)   at mx4j.tools.remote.caucho.CauchoServlet.invoke(CauchoServlet.java:139)        at com.cognos.pogo.mo
nitoring.jmx.remote.CognosHessianServlet.invoke(CognosHessianServlet.java:77)   at mx4j.tools.remote.caucho.CauchoServlet.ser
vice(CauchoServlet.java:71)     at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at com.cognos.pogo.isolation.
ServletWrapper.service(ServletWrapper.java:258) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applicati
onFilterChain.java:290) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)    at or
g.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)       at org.apache.catalina.core.StandardC
ontextValve.invoke(StandardContextValve.java:191)       at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValv
e.java:128)     at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)        at org.apache.catalin
a.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(Coyote
Adapter.java:286)       at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:852)   at org.apache.coyote.
http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)  at org.apache.tomcat.util.net.JIoEndpoint$Wor
ker.run(JIoEndpoint.java:447)   at java.lang.Thread.run(Unknown Source)

[hawaii_desperado]

The pogo log also has errors as well
pogo_2011-07-15.log:    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
pogo_2011-07-15.log:2011-07-15 12:57:47.515 ERROR [hreshold.ConsolidatedHealthMBeanCollection] pool-1-thread-1: Exception cre
ating proxy health MBean for com.cognos:type=Metrics,group=System,service=agentService java.lang.SecurityException

[hawaii_desperado]

Working with support we finally resolved the issues.  There were a few.  The main cause seems to be we had somehow missconfigured the various keysstore and sharing passwords.  It's surprising that nothing is written to the logs when these passwords are not configured correctly but exporting our configurations and comparing the passwords we found the mis-matches and corrected it.

Finally, the gateway tier had SELinux enabled and it was blocking port 9300 to the dispatcher.  I hope this information helps someone else in the future solve a problem quicker.