项目中再使用springcloud gateway做微服务网关，在线上的环境忽然发现有时候接口会出现卡顿，更多集中在登陆的时候，而且卡顿呈现不确定性再检查springcloud gateway的日志时候发现出现了Connection链接的异常reactor.netty.channel.AbortedException: Connection has been closed，期间还发现一个问题，因为服务部署在k8s上有时候即使出现这个异常后，也不影响服务使用但是时间长了之后，就会导致请求卡顿严重时候还会出现请求处理失败，在网上搜索包括查看github的issue之后发现偏幅都太宽泛，并不聚焦没有太多实质性的制导，最终只能硬着头皮完整走一遍，并且讲结果记录下来方便以后查看。

场景复现

gateway日志中的异常信息

2021-12-10 19:18:27.776 ERROR 22734 --- [or-http-epoll-1] com.ops.config.AppConfig    : Web exception handing: Connection has been closed

reactor.netty.channel.AbortedException: Connection has been closed
	at reactor.netty.channel.ChannelOperationsHandler.discard(ChannelOperationsHandler.java:356)
	at reactor.netty.channel.ChannelOperationsHandler.drain(ChannelOperationsHandler.java:366)
	at reactor.netty.channel.ChannelOperationsHandler.handlerRemoved(ChannelOperationsHandler.java:210)
	at io.netty.channel.AbstractChannelHandlerContext.callHandlerRemoved(AbstractChannelHandlerContext.java:961)
	at io.netty.channel.DefaultChannelPipeline.callHandlerRemoved0(DefaultChannelPipeline.java:638)
	at io.netty.channel.DefaultChannelPipeline.destroyDown(DefaultChannelPipeline.java:887)
	at io.netty.channel.DefaultChannelPipeline.destroyUp(DefaultChannelPipeline.java:853)
	at io.netty.channel.DefaultChannelPipeline.destroy(DefaultChannelPipeline.java:845)
	at io.netty.channel.DefaultChannelPipeline.access$700(DefaultChannelPipeline.java:46)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelUnregistered(DefaultChannelPipeline.java:1390)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:178)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:164)
	at io.netty.channel.DefaultChannelPipeline.fireChannelUnregistered(DefaultChannelPipeline.java:830)
	at io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:835)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:333)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
	at java.lang.Thread.run(Thread.java:748)

2021-12-10 19:18:27.779 ERROR 22734 --- [or-http-epoll-1] o.s.w.s.adapter.HttpWebHandlerAdapter    : [26191cc8] Error [java.lang.UnsupportedOperationException] for HTTP GET "/auth/oauth/authorize?response_type=code&scope=read&client_id=00&redirect_uri=http%3A%2F%2F192.168.101.64%2Fapi%2Foauth2OAuth&state=%7B%22redirectUrl%22:%22http://192.168.101.64/1000/team/role", but ServerHttpResponse already committed (302 FOUND)

2021-12-11 13:05:57.278 ERROR 27785 --- [or-http-epoll-5] com.ops.config.AppConfig    : Web exception handing: Connection has been closed

reactor.netty.channel.AbortedException: Connection has been closed
	at reactor.netty.channel.ChannelOperationsHandler.discard(ChannelOperationsHandler.java:356)
	at reactor.netty.channel.ChannelOperationsHandler.drain(ChannelOperationsHandler.java:366)
	at reactor.netty.channel.ChannelOperationsHandler.handlerRemoved(ChannelOperationsHandler.java:210)
	at io.netty.channel.AbstractChannelHandlerContext.callHandlerRemoved(AbstractChannelHandlerContext.java:961)
	at io.netty.channel.DefaultChannelPipeline.callHandlerRemoved0(DefaultChannelPipeline.java:638)
	at io.netty.channel.DefaultChannelPipeline.destroyDown(DefaultChannelPipeline.java:887)
	at io.netty.channel.DefaultChannelPipeline.destroyUp(DefaultChannelPipeline.java:853)
	at io.netty.channel.DefaultChannelPipeline.destroy(DefaultChannelPipeline.java:845)
	at io.netty.channel.DefaultChannelPipeline.access$700(DefaultChannelPipeline.java:46)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelUnregistered(DefaultChannelPipeline.java:1390)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:178)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:164)
	at io.netty.channel.DefaultChannelPipeline.fireChannelUnregistered(DefaultChannelPipeline.java:830)
	at io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:835)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:333)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
	at java.lang.Thread.run(Thread.java:748)

2021-12-11 13:05:57.281 ERROR 27785 --- [or-http-epoll-5] o.s.w.s.adapter.HttpWebHandlerAdapter    : [8d36ff6a] Error [java.lang.UnsupportedOperationException] for HTTP POST "/sws/webhook/hws/alert", but ServerHttpResponse already committed (200 OK)

抛析问题

问题分析比较简单了，第一次碰到这个异常我也不认识他，他也不认识我话不多少直接上google查看一番，经过baidu和google之后，基本上就能确定了是因为netty本身的bug引起的，那么解决问题就简单了升级netty版本到修复了该问题的新版本就ok了。
官方bug链接：github.com/reactor/rea…

解决问题

既然问题确定了后那么就直接开始动手升级

1.升级netty到0.9.7RELEASE版本

通过查看官方在github中提供的信息，说到在netty的0.8.5版本中就进行了修复（但是实际结果中bug还是存在）。项目中实际使用的是springboot2.1.4.RELEASE + springcloud Greenwich.SR1自动引入的io.projectreactor.netty:reactor-netty:0.8.6.RELEASE版本。结合搜到的一些文章提到0.9.4版本修复了该问题，但是又因为0.9.4还有问题因此计划一次性升级到0.9.7修复缺陷。

方法1:单独升级reactor-netty到0.9.7版本 io.projectreactor.netty:reactor-netty:0.9.7.RELEASE
编译通过启动服务时候失败，因为版本差异太大导致其他依赖所需方法和类启动报错，不兼容低版本的springboot，因此决定通过整体升级springboot的方式来进行升级，但是springboot和springcloud的版本关联也比较大，因此需要做同步升级；

方法2:升级springboot到2.2.7.RELEASE版本，同时升级springcloud到Hoxton.SR12
通过查看springboot中的依赖然后找到对应的版本，具体步骤如下：

springboot第三方依赖版本查看

  1. spring-boot/spring-boot-project/spring-boot-dependencies/pom.xml查看引用的依赖版本: github.com/spring-proj…;
  2. release note 查看依赖版本变化：github.com/spring-proj…;
  3. netty版本查看：github.com/reactor/rea…
  4. 升级springcloud到对应的版本 start.spring.io/actuato… (浏览器打不开可以通过curl查看)；

相关链接：
springboot start工作原理：
blog.csdn.net/weixin_39…
jb51.net/article/188…

2.解决版本升级带来的其他问题

问题1:缺少tomcat依赖
升级springboot到2.2.7.RELEASE 和 springcloud Hoxton.SR12之后，启动服务提示缺少servlet.

处理方式：添加tomcat依赖 implementation 'org.springframework.boot:spring-boot-starter-tomcat'
相关链接： stackoverflow.com/quest…

问题2:0.9.7版本reactor-netty找不到方法，服务启动失败

***************************
APPLICATION FAILED TO START
***************************

Description:

An attempt was made to call a method that does not exist. The attempt was made from the following location:

    org.springframework.cloud.gateway.config.GatewayAutoConfiguration$NettyConfiguration.buildConnectionProvider(GatewayAutoConfiguration.java:798)

The following method did not exist:

    reactor.netty.resources.ConnectionProvider$Builder.evictInBackground(Ljava/time/Duration;)Lreactor/netty/resources/ConnectionProvider$ConnectionPoolSpec;

The method's class, reactor.netty.resources.ConnectionProvider$Builder, is available from the following locations:

    jar:file:/Users/qu/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty/0.9.7.RELEASE/3a63889c056a7da153d695726daa0614bc02bcf5/reactor-netty-0.9.7.RELEASE.jar!/reactor/netty/resources/ConnectionProvider$Builder.class

It was loaded from the following location:

    file:/Users/qu/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty/0.9.7.RELEASE/3a63889c056a7da153d695726daa0614bc02bcf5/reactor-netty-0.9.7.RELEASE.jar


Action:

Correct the classpath of your application so that it contains a single, compatible version of reactor.netty.resources.ConnectionProvider$Builder


Process finished with exit code 1

处理方式：升级reactor-netty版本到0.9.13.RELEASE
相关链接：
stackoom.com/question/4…
codenong.com/cs10704728…

问题3. 升级到0.9.13之后ES health Bean找不到服务启动失败
Failed to instantiate [org.springframework.boot.actuate.health.HealthContributor]: Factory method 'elasticsearchRestHealthContributor' threw exception; nested exception is java.lang.IllegalArgumentException: Beans must not be empty 找不到

Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'healthContributorRegistry' defined in class path resource [org/springframework/boot/actuate/autoconfigure/health/HealthEndpointConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.boot.actuate.health.HealthContributorRegistry]: Factory method 'healthContributorRegistry' threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'elasticsearchRestHealthContributor' defined in class path resource [org/springframework/boot/actuate/autoconfigure/elasticsearch/ElasticSearchRestHealthContributorAutoConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.boot.actuate.health.HealthContributor]: Factory method 'elasticsearchRestHealthContributor' threw exception; nested exception is java.lang.IllegalArgumentException: Beans must not be empty
	at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:656)
	at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:636)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1338)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1177)
	at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1306)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1226)
	at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:885)
	at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:789)
	... 55 common frames omitted
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.boot.actuate.health.HealthContributorRegistry]: Factory method 'healthContributorRegistry' threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'elasticsearchRestHealthContributor' defined in class path resource [org/springframework/boot/actuate/autoconfigure/elasticsearch/ElasticSearchRestHealthContributorAutoConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.boot.actuate.health.HealthContributor]: Factory method 'elasticsearchRestHealthContributor' threw exception; nested exception is java.lang.IllegalArgumentException: Beans must not be empty
	at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:185)
	at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:651)
	... 69 common frames omitted
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'elasticsearchRestHealthContributor' defined in class path resource [org/springframework/boot/actuate/autoconfigure/elasticsearch/ElasticSearchRestHealthContributorAutoConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.boot.actuate.health.HealthContributor]: Factory method 'elasticsearchRestHealthContributor' threw exception; nested exception is java.lang.IllegalArgumentException: Beans must not be empty
	at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:656)
	at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:636)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1338)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1177)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeansOfType(DefaultListableBeanFactory.java:623)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeansOfType(DefaultListableBeanFactory.java:611)
	at org.springframework.context.support.AbstractApplicationContext.getBeansOfType(AbstractApplicationContext.java:1242)
	at org.springframework.boot.actuate.autoconfigure.health.HealthEndpointConfiguration.healthContributorRegistry(HealthEndpointConfiguration.java:78)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154)
	... 70 common frames omitted
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.boot.actuate.health.HealthContributor]: Factory method 'elasticsearchRestHealthContributor' threw exception; nested exception is java.lang.IllegalArgumentException: Beans must not be empty
	at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:185)
	at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:651)
	... 88 common frames omitted
Caused by: java.lang.IllegalArgumentException: Beans must not be empty
	at org.springframework.util.Assert.notEmpty(Assert.java:549)
	at org.springframework.boot.actuate.autoconfigure.health.AbstractCompositeHealthContributorConfiguration.createContributor(AbstractCompositeHealthContributorConfiguration.java:52)
	at org.springframework.boot.actuate.autoconfigure.elasticsearch.ElasticSearchRestHealthContributorAutoConfiguration.elasticsearchRestHealthContributor(ElasticSearchRestHealthContributorAutoConfiguration.java:55)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154)
	... 89 common frames omitted

处理方式1: 直接关闭设置management.health.elasticsearch.enabled=false，问题解决但是过于暴力不推荐使用。
相关链接： github.com/spring-proj…

处理方式2: 配置Elasticsearch high level客户端Bean并且配置management.health.elasticsearch.response-timeout，配置完成后问题解决。

@Bean
public RestHighLevelClient client() {
    //todo 生成host的逻辑根据当前es host配置方式转换，线上就不需要修改ES配置
    List hostList = Arrays.stream(hosts.split(Consts.COMMA)).map(h -> new HttpHost(h, port, httpType)).collect(Collectors.toList());
    HttpHost[] hosts = new HttpHost[hostList.size()];
    hostList.toArray(hosts);
    RestClientBuilder builder = RestClient.builder(hosts);
    // 异步httpclient连接延时配置
    builder.setRequestConfigCallback(new RestClientBuilder.RequestConfigCallback() {
        @Override
        public RequestConfig.Builder customizeRequestConfig(RequestConfig.Builder requestConfigBuilder) {
            requestConfigBuilder.setConnectTimeout(RestClientBuilder.DEFAULT_CONNECT_TIMEOUT_MILLIS);
            requestConfigBuilder.setSocketTimeout(RestClientBuilder.DEFAULT_SOCKET_TIMEOUT_MILLIS);
            requestConfigBuilder.setConnectionRequestTimeout(RestClientBuilder.DEFAULT_CONNECT_TIMEOUT_MILLIS);
            return requestConfigBuilder;
        }
    });
    // 异步httpclient连接数配置
    builder.setHttpClientConfigCallback(new RestClientBuilder.HttpClientConfigCallback() {
        @Override
        public HttpAsyncClientBuilder customizeHttpClient(HttpAsyncClientBuilder httpClientBuilder) {
            httpClientBuilder.setMaxConnTotal(RestClientBuilder.DEFAULT_MAX_CONN_TOTAL);
            httpClientBuilder.setMaxConnPerRoute(RestClientBuilder.DEFAULT_MAX_CONN_PER_ROUTE);
            return httpClientBuilder;
        }
    });
    RestHighLevelClient client = new RestHighLevelClient(builder);
    return client;
}

bootstrap.yml

management:
  health:
    elasticsearch:
      response-timeout: 1000ms # 增加超时时间，避免不必要的检查失败

启动服务发现服务启动成功，并且服务调用成功。(如果服务启动成功遇到redis反序列化问题，参考链接)

总结

1.在spring cloud gateway中使用reactor-netty如果遇到reactor.netty.channel.AbortedException: Connection has been closed需要考虑升级netty版本；
2.对于依赖版本的修改会引起很多其他新问题，但是无论是新旧问题那种问题修改，都需要结合官方的文档和资料作为更新依据；

作者：troyqu
链接： juejin.cn/post/70436214

# java # spring boot # spring # bug # linux
Logo
华为云开发者联盟

为开发者提供学习成长、分享交流、生态实践、资源工具等服务，帮助开发者快速成长。

更多推荐

cover

实践探讨Python如何进行异常处理与日志记录

avatar 华为云开发者联盟
cover

HCDG天津站精彩回顾 | AI高效开发， ModelArts技术动手工作坊

avatar 华为云开发者联盟
cover

一次故障演练，十分钟自动搞定?

avatar 华为云开发者联盟