Tomcat启动卡住问题排查

mysql作为目前常用的数据库，有些问题排查还是要考虑进来的。

##问题简述

17年左右开发的一个老系统，使用jdk1.8，tomcat7发布的一个系统。架构使用stuts+Hibernet编写的项目，前段时间因为集团安全要求加固，业务偶发造成数据库报错连接异常，需要重启应用释放连接，但是这次竟然无法启动成功了，由此造成业务系统全部停止了。。。囧~~~

tomcat日志：

catalina.out日志信息
Jul 17, 2022 1:58:49 PM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deploying web application directory /usr/local/apache-tomcat/tomcat6/webapps/pda
Jul 17, 2022 1:58:49 PM org.apache.catalina.loader.WebappClassLoader validateJarFile
INFO: validateJarFile(/usr/local/apache-tomcat/tomcat6/webapps/pda/WEB-INF/lib/geronimo-servlet_3.0_spec-1.0.jar) - jar not loaded. See Servlet Spec 3.0, section 10.7.2. Offending class: javax/servlet/Servlet.class
Jul 17, 2022 1:58:49 PM org.apache.catalina.loader.WebappClassLoader validateJarFile
INFO: validateJarFile(/usr/local/apache-tomcat/tomcat6/webapps/pda/WEB-INF/lib/servlet-api.jar) - jar not loaded. See Servlet Spec 3.0, section 10.7.2. Offending class: javax/servlet/Servlet.class
Jul 17, 2022 1:58:56 PM org.apache.cxf.service.factory.ReflectionServiceFactoryBean buildServiceFromClass
INFO: Creating Service {http://www.example.org/ReturnPolicyNo/}ReturnPolicyNoService from class com.pda.gsc.ws.cxfservice.ReturnPolicyNo
Jul 17, 2022 1:58:57 PM org.apache.cxf.endpoint.ServerImpl initDestination
INFO: Setting the server's publish address to be /PolicyNo
Jul 17, 2022 1:58:57 PM org.apache.cxf.service.factory.ReflectionServiceFactoryBean buildServiceFromWSDL
INFO: Creating Service {http://docs.oasis-open.org/ws-dd/ns/discovery/2009/01}Discovery from WSDL: classpath:/org/apache/cxf/ws/discovery/wsdl/wsdd-discovery-1.1-wsdl-os.wsdl
Jul 17, 2022 1:58:57 PM org.apache.cxf.endpoint.ServerImpl initDestination
INFO: Setting the server's publish address to be soap.udp://239.255.255.250:3702
Jul 17, 2022 1:58:57 PM org.apache.cxf.service.factory.ReflectionServiceFactoryBean buildServiceFromClass
INFO: Creating Service {http://docs.oasis-open.org/ws-dd/ns/discovery/2009/01}DiscoveryProxy from class org.apache.cxf.jaxws.support.DummyImpl
Jul 17, 2022 1:58:57 PM org.apache.cxf.service.factory.ReflectionServiceFactoryBean buildServiceFromClass
INFO: Creating Service {http://come.enn.cn/CcsToPda/}CcsToPdaService from class com.pda.gsc.ws.ccsservice.CcsToPda
Jul 17, 2022 1:58:57 PM org.apache.cxf.endpoint.ServerImpl initDestination
INFO: Setting the server's publish address to be /Ccssendorder
Jul 17, 2022 1:58:57 PM org.apache.cxf.service.factory.ReflectionServiceFactoryBean buildServiceFromClass
INFO: Creating Service {http://www.enn.cn/InsuranceReturnInfo/}InsuranceReturnInfoService from class com.pda.gsc.ws.insuranceservice.InsuranceReturnInfo
Jul 17, 2022 1:58:57 PM org.apache.cxf.endpoint.ServerImpl initDestination
INFO: Setting the server's publish address to be /InsuranceReturnInfo
Jul 17, 2022 1:59:50 PM org.apache.catalina.core.AprLifecycleListener init
INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: /usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
Jul 17, 2022 1:59:50 PM org.apache.catalina.startu

###尝试排查

1.  nginx转发、网络转发

        出现此类问题的时候，以为是nginx转发出现问题，未进行转发数据。随后联系了服务器组与网路组排查告知应该是应用自身问题。~~~

2.  应用排查

        先验证tomcat 是否正常，将应用从tomcat中移除，进行单独启动tomcat，发现tomcat可以正常启动。(中间也试过换tomcat8单独启动也是正常)~~~~

        应用一直没有做为任何修改，不应该是应用的问题。为了验证，将应用在本地启动，一切正常。之后怀疑是数据库的问题，使用Navicat工具发现也链接正常。之后又从正式环境找了一台服务器部署应用启动，发现也启动正常。

3. 端口排查

        验证了应用可以链接的所有外部端口，均可以正常访问。之后联系网络组告知封闭了443与80端口，是不是此问题导致的。放开之后重启也是如此。

解决问题过程中，baidu了不少文章，做一个参考。修改之后，错误依旧。

Tomcat启动一直卡在这里，无任何报错和任何输出

解决：

找到jdk1.x.x_xx/jre/lib/security/Java.security文件，在文件中找到securerandom.source这个设置项，将其改为：

securerandom.source=file:/dev/./urandom

原因：

linux或者部分unix系统提供随机数设备是/dev/random 和/dev/urandom ，

两个有区别，urandom安全性没有random高，但random需要时间间隔生成随机数。jdk默认调用random。

然后就很简单啦，找到对应的配置文件去修改就好了

最后使用jstack 进行监控进程ID，进行排查，发现jdbc卡在RUNNABLE状态，导致有请求，无返回。但是使用Navicat可以正常链接，抱着试一试的问题，咨询DBA，告知也是正常。我就把这个错误信息发给了他。数据库排查发现 Waiting for table flush，确实有问题，因为最近在做数据库不备份，参数有问题，需要调整。之后重启应用解决。

困扰了2天的问题，终于解决。排查过程中真的是自我怀疑呀。

 jstack -l 60735 >> thread.txt

thread.txt信息"localhost-startStop-1" #14 daemon prio=5 os_prio=0 tid=0x00007fc208006000 nid=0xed5d runnable [0x00007fc21626a000]java.lang.Thread.State: RUNNABLEat java.net.SocketInputStream.socketRead0(Native Method)at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)at java.net.SocketInputStream.read(SocketInputStream.java:171)at java.net.SocketInputStream.read(SocketInputStream.java:141)at com.mysql.jdbc.util.ReadAheadInputStream.fill(ReadAheadInputStream.java:114)at com.mysql.jdbc.util.ReadAheadInputStream.readFromUnderlyingStreamIfNecessary(ReadAheadInputStream.java:161)at com.mysql.jdbc.util.ReadAheadInputStream.read(ReadAheadInputStream.java:189)- locked <0x00000006c7813408> (a com.mysql.jdbc.util.ReadAheadInputStream)at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3163)at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3620)at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3609)at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4160)at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2617)at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2778)at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2828)- locked <0x00000006c77fbd50> (a com.mysql.jdbc.JDBC4Connection)at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2777)at com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1651)- locked <0x00000006c77fbd50> (a com.mysql.jdbc.JDBC4Connection)at com.mysql.jdbc.DatabaseMetaData.extractForeignKeyFromCreateTable(DatabaseMetaData.java:1416)at com.mysql.jdbc.DatabaseMetaData$5.forEach(DatabaseMetaData.java:3596)at com.mysql.jdbc.DatabaseMetaData$5.forEach(DatabaseMetaData.java:3585)at com.mysql.jdbc.IterateBlock.doForAll(IterateBlock.java:50)at com.mysql.jdbc.DatabaseMetaData.getImportedKeys(DatabaseMetaData.java:3584)at com.mchange.v2.c3p0.impl.NewProxyDatabaseMetaData.getImportedKeys(NewProxyDatabaseMetaData.java:3062)at org.hibernate.tool.hbm2ddl.TableMetadata.initForeignKeys(TableMetadata.java:162)at org.hibernate.tool.hbm2ddl.TableMetadata.(TableMetadata.java:60)at org.hibernate.tool.hbm2ddl.DatabaseMetadata.getTableMetadata(DatabaseMetadata.java:114)at org.hibernate.cfg.Configuration.generateSchemaUpdateScript(Configuration.java:1133)at org.hibernate.tool.hbm2ddl.SchemaUpdate.execute(SchemaUpdate.java:212)at org.hibernate.tool.hbm2ddl.SchemaUpdate.execute(SchemaUpdate.java:178)at org.hibernate.internal.SessionFactoryImpl.(SessionFactoryImpl.java:503)at org.hibernate.cfg.Configuration.buildSessionFactory(Configuration.java:1750)at org.hibernate.cfg.Configuration.buildSessionFactory(Configuration.java:1788)at org.springframework.orm.hibernate4.LocalSessionFactoryBuilder.buildSessionFactory(LocalSessionFactoryBuilder.java:247)at org.springframework.orm.hibernate4.LocalSessionFactoryBean.buildSessionFactory(LocalSessionFactoryBean.java:373)at org.springframework.orm.hibernate4.LocalSessionFactoryBean.afterPropertiesSet(LocalSessionFactoryBean.java:358)at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1571)at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1509)at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:521)at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:458)at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:296)at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:223)- locked <0x00000006c6f2fe48> (a java.util.concurrent.ConcurrentHashMap)at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:293)at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:610)at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:932)at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:479)- locked <0x00000006c6e41860> (a java.lang.Object)at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:389)at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:294)at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:112)at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4973)at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5467)- locked <0x00000006c633a900> (a org.apache.catalina.core.StandardContext)at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)- locked <0x00000006c633a900> (a org.apache.catalina.core.StandardContext)at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:901)at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:877)at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:632)at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1229)at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:1875)at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)at java.util.concurrent.FutureTask.run(FutureTask.java:266)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)at java.lang.Thread.run(Thread.java:745)

数据库备份错误信息：29864832 root localhost  Query 53084 Waiting for table flush FLUSH /*!40101 LOCAL */ TABLES
29846815 root localhost pda Query 65697 Sending data SELECT /*!40001 SQL_NO_CACHE */ * FROM `PoolLogs`

 This is Fine

本文来自互联网用户投稿，文章观点仅代表作者本人，不代表本站立场，不承担相关法律责任。如若转载，请注明出处。 如若内容造成侵权/违法违规/事实不符，请点击【内容举报】进行投诉反馈！