项目中再使用springcloud gateway做微服务网关,在线上的环境忽然发现有时候接口会出现卡顿,更多集中在登陆的时候,而且卡顿呈现不确定性再检查springcloud gateway的日志时候发现出现了Connection链接的异常reactor.netty.channel.AbortedException: Connection has been closed,期间还发现一个问题,因为服务部署在k8s上有时候即使出现这个异常后,也不影响服务使用但是时间长了之后,就会导致请求卡顿严重时候还会出现请求处理失败,在网上搜索包括查看github的issue之后发现偏幅都太宽泛,并不聚焦没有太多实质性的制导,最终只能硬着头皮完整走一遍,并且讲结果记录下来方便以后查看。

场景复现

gateway日志中的异常信息

2021-12-10 19:18:27.776 ERROR 22734 --- [or-http-epoll-1] com.ops.config.AppConfig    : Web exception handing: Connection has been closed

reactor.netty.channel.AbortedException: Connection has been closed
	at reactor.netty.channel.ChannelOperationsHandler.discard(ChannelOperationsHandler.java:356)
	at reactor.netty.channel.ChannelOperationsHandler.drain(ChannelOperationsHandler.java:366)
	at reactor.netty.channel.ChannelOperationsHandler.handlerRemoved(ChannelOperationsHandler.java:210)
	at io.netty.channel.AbstractChannelHandlerContext.callHandlerRemoved(AbstractChannelHandlerContext.java:961)
	at io.netty.channel.DefaultChannelPipeline.callHandlerRemoved0(DefaultChannelPipeline.java:638)
	at io.netty.channel.DefaultChannelPipeline.destroyDown(DefaultChannelPipeline.java:887)
	at io.netty.channel.DefaultChannelPipeline.destroyUp(DefaultChannelPipeline.java:853)
	at io.netty.channel.DefaultChannelPipeline.destroy(DefaultChannelPipeline.java:845)
	at io.netty.channel.DefaultChannelPipeline.access$700(DefaultChannelPipeline.java:46)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelUnregistered(DefaultChannelPipeline.java:1390)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:178)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:164)
	at io.netty.channel.DefaultChannelPipeline.fireChannelUnregistered(DefaultChannelPipeline.java:830)
	at io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:835)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:333)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
	at java.lang.Thread.run(Thread.java:748)

2021-12-10 19:18:27.779 ERROR 22734 --- [or-http-epoll-1] o.s.w.s.adapter.HttpWebHandlerAdapter    : [26191cc8] Error [java.lang.UnsupportedOperationException] for HTTP GET "/auth/oauth/authorize?response_type=code&scope=read&client_id=00&redirect_uri=http%3A%2F%2F192.168.101.64%2Fapi%2Foauth2OAuth&state=%7B%22redirectUrl%22:%22http://192.168.101.64/1000/team/role", but ServerHttpResponse already committed (302 FOUND)

2021-12-11 13:05:57.278 ERROR 27785 --- [or-http-epoll-5] com.ops.config.AppConfig    : Web exception handing: Connection has been closed

reactor.netty.channel.AbortedException: Connection has been closed
	at reactor.netty.channel.ChannelOperationsHandler.discard(ChannelOperationsHandler.java:356)
	at reactor.netty.channel.ChannelOperationsHandler.drain(ChannelOperationsHandler.java:366)
	at reactor.netty.channel.ChannelOperationsHandler.handlerRemoved(ChannelOperationsHandler.java:210)
	at io.netty.channel.AbstractChannelHandlerContext.callHandlerRemoved(AbstractChannelHandlerContext.java:961)
	at io.netty.channel.DefaultChannelPipeline.callHandlerRemoved0(DefaultChannelPipeline.java:638)
	at io.netty.channel.DefaultChannelPipeline.destroyDown(DefaultChannelPipeline.java:887)
	at io.netty.channel.DefaultChannelPipeline.destroyUp(DefaultChannelPipeline.java:853)
	at io.netty.channel.DefaultChannelPipeline.destroy(DefaultChannelPipeline.java:845)
	at io.netty.channel.DefaultChannelPipeline.access$700(DefaultChannelPipeline.java:46)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelUnregistered(DefaultChannelPipeline.java:1390)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:178)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:164)
	at io.netty.channel.DefaultChannelPipeline.fireChannelUnregistered(DefaultChannelPipeline.java:830)
	at io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:835)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:333)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
	at java.lang.Thread.run(Thread.java:748)

2021-12-11 13:05:57.281 ERROR 27785 --- [or-http-epoll-5] o.s.w.s.adapter.HttpWebHandlerAdapter    : [8d36ff6a] Error [java.lang.UnsupportedOperationException] for HTTP POST "/sws/webhook/hws/alert", but ServerHttpResponse already committed (200 OK)

抛析问题

问题分析比较简单了,第一次碰到这个异常我也不认识他,他也不认识我话不多少直接上google查看一番,经过baidu和google之后,基本上就能确定了是因为netty本身的bug引起的,那么解决问题就简单了升级netty版本到修复了该问题的新版本就ok了。
官方bug链接:github.com/reactor/rea

解决问题

既然问题确定了后那么就直接开始动手升级

1.升级netty到0.9.7RELEASE版本

通过查看官方在github中提供的信息,说到在netty的0.8.5版本中就进行了修复(但是实际结果中bug还是存在)。项目中实际使用的是springboot2.1.4.RELEASE + springcloud Greenwich.SR1自动引入的io.projectreactor.netty:reactor-netty:0.8.6.RELEASE版本。结合搜到的一些文章提到0.9.4版本修复了该问题,但是又因为0.9.4还有问题因此计划一次性升级到0.9.7修复缺陷。

方法1:单独升级reactor-netty到0.9.7版本 io.projectreactor.netty:reactor-netty:0.9.7.RELEASE
编译通过启动服务时候失败,因为版本差异太大导致其他依赖所需方法和类启动报错,不兼容低版本的springboot,因此决定通过整体升级springboot的方式来进行升级,但是springboot和springcloud的版本关联也比较大,因此需要做同步升级;

方法2:升级springboot到2.2.7.RELEASE版本,同时升级springcloud到Hoxton.SR12
通过查看springboot中的依赖然后找到对应的版本,具体步骤如下:

springboot第三方依赖版本查看

  1. spring-boot/spring-boot-project/spring-boot-dependencies/pom.xml查看引用的依赖版本: github.com/spring-proj…;
  2. release note 查看依赖版本变化:github.com/spring-proj…;
  3. netty版本查看:github.com/reactor/rea
  4. 升级springcloud到对应的版本 start.spring.io/actuato… (浏览器打不开可以通过curl查看);

相关链接:
springboot start工作原理:
blog.csdn.net/weixin_39
jb51.net/article/188

2.解决版本升级带来的其他问题

问题1:缺少tomcat依赖
升级springboot到2.2.7.RELEASE 和 springcloud Hoxton.SR12之后,启动服务提示缺少servlet.

处理方式:添加tomcat依赖 implementation 'org.springframework.boot:spring-boot-starter-tomcat'
相关链接stackoverflow.com/quest

问题2:0.9.7版本reactor-netty找不到方法,服务启动失败

***************************
APPLICATION FAILED TO START
***************************

Description:

An attempt was made to call a method that does not exist. The attempt was made from the following location:

    org.springframework.cloud.gateway.config.GatewayAutoConfiguration$NettyConfiguration.buildConnectionProvider(GatewayAutoConfiguration.java:798)

The following method did not exist:

    reactor.netty.resources.ConnectionProvider$Builder.evictInBackground(Ljava/time/Duration;)Lreactor/netty/resources/ConnectionProvider$ConnectionPoolSpec;

The method's class, reactor.netty.resources.ConnectionProvider$Builder, is available from the following locations:

    jar:file:/Users/qu/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty/0.9.7.RELEASE/3a63889c056a7da153d695726daa0614bc02bcf5/reactor-netty-0.9.7.RELEASE.jar!/reactor/netty/resources/ConnectionProvider$Builder.class

It was loaded from the following location:

    file:/Users/qu/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty/0.9.7.RELEASE/3a63889c056a7da153d695726daa0614bc02bcf5/reactor-netty-0.9.7.RELEASE.jar


Action:

Correct the classpath of your application so that it contains a single, compatible version of reactor.netty.resources.ConnectionProvider$Builder


Process finished with exit code 1

处理方式:升级reactor-netty版本到0.9.13.RELEASE
相关链接
stackoom.com/question/4
codenong.com/cs10704728

问题3. 升级到0.9.13之后ES health Bean找不到服务启动失败
Failed to instantiate [org.springframework.boot.actuate.health.HealthContributor]: Factory method 'elasticsearchRestHealthContributor' threw exception; nested exception is java.lang.IllegalArgumentException: Beans must not be empty 找不到

Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'healthContributorRegistry' defined in class path resource [org/springframework/boot/actuate/autoconfigure/health/HealthEndpointConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.boot.actuate.health.HealthContributorRegistry]: Factory method 'healthContributorRegistry' threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'elasticsearchRestHealthContributor' defined in class path resource [org/springframework/boot/actuate/autoconfigure/elasticsearch/ElasticSearchRestHealthContributorAutoConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.boot.actuate.health.HealthContributor]: Factory method 'elasticsearchRestHealthContributor' threw exception; nested exception is java.lang.IllegalArgumentException: Beans must not be empty
	at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:656)
	at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:636)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1338)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1177)
	at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1306)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1226)
	at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:885)
	at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:789)
	... 55 common frames omitted
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.boot.actuate.health.HealthContributorRegistry]: Factory method 'healthContributorRegistry' threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'elasticsearchRestHealthContributor' defined in class path resource [org/springframework/boot/actuate/autoconfigure/elasticsearch/ElasticSearchRestHealthContributorAutoConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.boot.actuate.health.HealthContributor]: Factory method 'elasticsearchRestHealthContributor' threw exception; nested exception is java.lang.IllegalArgumentException: Beans must not be empty
	at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:185)
	at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:651)
	... 69 common frames omitted
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'elasticsearchRestHealthContributor' defined in class path resource [org/springframework/boot/actuate/autoconfigure/elasticsearch/ElasticSearchRestHealthContributorAutoConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.boot.actuate.health.HealthContributor]: Factory method 'elasticsearchRestHealthContributor' threw exception; nested exception is java.lang.IllegalArgumentException: Beans must not be empty
	at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:656)
	at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:636)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1338)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1177)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeansOfType(DefaultListableBeanFactory.java:623)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeansOfType(DefaultListableBeanFactory.java:611)
	at org.springframework.context.support.AbstractApplicationContext.getBeansOfType(AbstractApplicationContext.java:1242)
	at org.springframework.boot.actuate.autoconfigure.health.HealthEndpointConfiguration.healthContributorRegistry(HealthEndpointConfiguration.java:78)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154)
	... 70 common frames omitted
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.boot.actuate.health.HealthContributor]: Factory method 'elasticsearchRestHealthContributor' threw exception; nested exception is java.lang.IllegalArgumentException: Beans must not be empty
	at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:185)
	at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:651)
	... 88 common frames omitted
Caused by: java.lang.IllegalArgumentException: Beans must not be empty
	at org.springframework.util.Assert.notEmpty(Assert.java:549)
	at org.springframework.boot.actuate.autoconfigure.health.AbstractCompositeHealthContributorConfiguration.createContributor(AbstractCompositeHealthContributorConfiguration.java:52)
	at org.springframework.boot.actuate.autoconfigure.elasticsearch.ElasticSearchRestHealthContributorAutoConfiguration.elasticsearchRestHealthContributor(ElasticSearchRestHealthContributorAutoConfiguration.java:55)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154)
	... 89 common frames omitted

处理方式1: 直接关闭设置management.health.elasticsearch.enabled=false,问题解决但是过于暴力不推荐使用。
相关链接github.com/spring-proj

处理方式2: 配置Elasticsearch high level客户端Bean并且配置management.health.elasticsearch.response-timeout,配置完成后问题解决。

@Bean
public RestHighLevelClient client() {
    //todo 生成host的逻辑根据当前es host配置方式转换,线上就不需要修改ES配置
    List hostList = Arrays.stream(hosts.split(Consts.COMMA)).map(h -> new HttpHost(h, port, httpType)).collect(Collectors.toList());
    HttpHost[] hosts = new HttpHost[hostList.size()];
    hostList.toArray(hosts);
    RestClientBuilder builder = RestClient.builder(hosts);
    // 异步httpclient连接延时配置
    builder.setRequestConfigCallback(new RestClientBuilder.RequestConfigCallback() {
        @Override
        public RequestConfig.Builder customizeRequestConfig(RequestConfig.Builder requestConfigBuilder) {
            requestConfigBuilder.setConnectTimeout(RestClientBuilder.DEFAULT_CONNECT_TIMEOUT_MILLIS);
            requestConfigBuilder.setSocketTimeout(RestClientBuilder.DEFAULT_SOCKET_TIMEOUT_MILLIS);
            requestConfigBuilder.setConnectionRequestTimeout(RestClientBuilder.DEFAULT_CONNECT_TIMEOUT_MILLIS);
            return requestConfigBuilder;
        }
    });
    // 异步httpclient连接数配置
    builder.setHttpClientConfigCallback(new RestClientBuilder.HttpClientConfigCallback() {
        @Override
        public HttpAsyncClientBuilder customizeHttpClient(HttpAsyncClientBuilder httpClientBuilder) {
            httpClientBuilder.setMaxConnTotal(RestClientBuilder.DEFAULT_MAX_CONN_TOTAL);
            httpClientBuilder.setMaxConnPerRoute(RestClientBuilder.DEFAULT_MAX_CONN_PER_ROUTE);
            return httpClientBuilder;
        }
    });
    RestHighLevelClient client = new RestHighLevelClient(builder);
    return client;
}

bootstrap.yml

management:
  health:
    elasticsearch:
      response-timeout: 1000ms # 增加超时时间,避免不必要的检查失败

启动服务发现服务启动成功,并且服务调用成功。(如果服务启动成功遇到redis反序列化问题,参考链接)

总结

1.在spring cloud gateway中使用reactor-netty如果遇到reactor.netty.channel.AbortedException: Connection has been closed需要考虑升级netty版本;
2.对于依赖版本的修改会引起很多其他新问题,但是无论是新旧问题那种问题修改,都需要结合官方的文档和资料作为更新依据;

作者:troyqu
链接: juejin.cn/post/70436214

Logo

华为云1024程序员节送福利,参与活动赢单人4000元礼包,更有热门技术干货免费学习

更多推荐