redis

Redis自动断开连接错误排查与修复

从Redis自动断开连接看配置的正确使用方式

Posted by     "Connor" on Monday, November 18, 2019

TOC

异常信息

在使用spring-boot-starter-cache和spring-boot-starter-data-redis添加了redis缓存之后，程序间断的报以下异常。但是重复请求之后又能正常请求到缓存数据。

org.springframework.data.redis.RedisConnectionFailureException: Unexpected end of stream.; nested exception is redis.clients.jedis.exceptions.JedisConnectionException: Unexpected end of stream.
    at org.springframework.data.redis.connection.jedis.JedisExceptionConverter.convert(JedisExceptionConverter.java:67)
    at org.springframework.data.redis.connection.jedis.JedisExceptionConverter.convert(JedisExceptionConverter.java:41)
    at org.springframework.data.redis.PassThroughExceptionTranslationStrategy.translate(PassThroughExceptionTranslationStrategy.java:37)
    at org.springframework.data.redis.FallbackExceptionTranslationStrategy.translate(FallbackExceptionTranslationStrategy.java:37)
    at org.springframework.data.redis.connection.jedis.JedisConnection.convertJedisAccessException(JedisConnection.java:241)
    at org.springframework.data.redis.connection.jedis.JedisConnection.exists(JedisConnection.java:802)
    at org.springframework.data.redis.connection.DefaultStringRedisConnection.exists(DefaultStringRedisConnection.java:264)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springframework.data.redis.core.CloseSuppressingInvocationHandler.invoke(CloseSuppressingInvocationHandler.java:57)
    at com.sun.proxy.$Proxy168.exists
    at org.springframework.data.redis.cache.RedisCache$AbstractRedisCacheCallback.waitForLock(RedisCache.java:614)
    at org.springframework.data.redis.cache.RedisCache$AbstractRedisCacheCallback.doInRedis(RedisCache.java:577)
    at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:207)
    at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:169)
    at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:157)
    at org.springframework.data.redis.cache.RedisCache.evict(RedisCache.java:280)
    at org.springframework.data.redis.cache.RedisCache.evict(RedisCache.java:269)
    at org.springframework.cache.interceptor.AbstractCacheInvoker.doEvict(AbstractCacheInvoker.java:98)
    at org.springframework.cache.interceptor.CacheAspectSupport.performCacheEvict(CacheAspectSupport.java:477)
    at org.springframework.cache.interceptor.CacheAspectSupport.processCacheEvicts(CacheAspectSupport.java:460)
    at org.springframework.cache.interceptor.CacheAspectSupport.execute(CacheAspectSupport.java:418)
    at org.springframework.cache.interceptor.CacheAspectSupport.execute(CacheAspectSupport.java:324)
    at org.springframework.cache.interceptor.CacheInterceptor.invoke(CacheInterceptor.java:61)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
    at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:671)
    ...
    at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:736)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
    at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:52)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:168)
    at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
    at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:671)
    at com.yzw.ibuild.cscec8b.controller.HomeController$$EnhancerBySpringCGLIB$$6a4971c0.saveCostManager(<generated>)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
    at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
    at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
    at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
    at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:661)
    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:111)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:109)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:93)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:106)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:496)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
    at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:677)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803)
    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:790)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1468)
    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:748)

排查过程

注意到是JedisConnectionException所以推测是redis连接断开导致。检查redis的连接配置

项目使用spring-boot-starter-data-redis集成redis 版本1.5.14，即Redis客户端为Jedis+连接池

spring.redis.database=2
spring.redis.pool.max-idle=8
spring.redis.pool.min-idle=1
spring.redis.pool.max-active=8
spring.redis.timeout=3000
spring.redis.sentinel.master=mymaster
spring.redis.sentinel.nodes=<SentinelNodesIPs>

发现其中并未有空闲连接自动检测的配置，推测是因为Redis设置了某个配置在客户端一段时间为使用连接时自动关闭该客户端的连接。而客户端在使用连接时并未对该连接的有效性做检查，导致上述异常。

解决过程

客户端配置

  • 默认配置

    public class JedisPoolConfig extends GenericObjectPoolConfig {
public JedisPoolConfig() {
    this.setTestWhileIdle(true);
    this.setMinEvictableIdleTimeMillis(60000L);
    this.setTimeBetweenEvictionRunsMillis(30000L);
    this.setNumTestsPerEvictionRun(-1);
}
}

  • 添加空闲连接检测相关配置

    spring.redis.pool.testOnBorrow=true
spring.redis.pool.testWhileIdle=true
spring.redis.pool.timeBetweenEvictionRunsMillis=30000

因为spring-boot-starter-data-redis并未提供以上三个配置的自动配置，所以需要自己使用代码配置JedisConnectionFactory的方式来设置。

@Configuration
public class RedisConfig {

    @Autowired
    private RedisProperties redisProperties;

    @Value("${spring.redis.pool.testOnBorrow}")
    private boolean testOnBorrow;

    @Value("${spring.redis.pool.testWhileIdle}")
    private boolean testWhileIdle;

    @Value("${spring.redis.pool.timeBetweenEvictionRunsMillis}")
    private long timeBetweenEvictionRunsMillis;

    @Bean
    public JedisPoolConfig jedisPoolConfig() {
        JedisPoolConfig config = new JedisPoolConfig();
        config.setMaxIdle(redisProperties.getPool().getMaxIdle());
        config.setMinIdle(redisProperties.getPool().getMinIdle());
        config.setMaxTotal(redisProperties.getPool().getMaxActive());
        config.setTestOnBorrow(testOnBorrow);
        config.setTestWhileIdle(testWhileIdle);
        config.setTimeBetweenEvictionRunsMillis(timeBetweenEvictionRunsMillis);
        return config;
    }

    @Bean
    public RedisSentinelConfiguration redisSentinelConfiguration() {
        RedisSentinelConfiguration redisSentinelConfiguration = new RedisSentinelConfiguration();
        redisSentinelConfiguration.setMaster(redisProperties.getSentinel().getMaster());
        Set<RedisNode> redisNodes = Arrays.stream(redisProperties.getSentinel().getNodes().split(",")).map(nodeStr -> {
            String[] split = nodeStr.split(":");
            return new RedisNode(split[0], Integer.valueOf(split[1]));
        }).collect(Collectors.toSet());
        redisSentinelConfiguration.setSentinels(redisNodes);
        return redisSentinelConfiguration;
    }

    @Bean
    public JedisConnectionFactory jedisConnectionFactory(JedisPoolConfig config, RedisSentinelConfiguration redisSentinelConfiguration) {
        JedisConnectionFactory factory = new JedisConnectionFactory(redisSentinelConfiguration, config);
        factory.setDatabase(redisProperties.getDatabase());
        factory.setTimeout(redisProperties.getTimeout());
        return factory;
    }
}

Redis服务配置

服务器端主动关闭连接相关的参数有：

client-output-buffer-limit

对于Redis服务器的输出（也就是命令的返回值）来说，其大小通常是不可控制的。有可能一个简单的命令，能够产生体积庞大的返回数据。另外也有可能因为执行了太多命令，导致产生返回数据的速率超过了往客户端发送的速率，这是也会导致服务器堆积大量消息，从而导致输出缓冲区越来越大，占用过多内存，甚至导致系统崩溃。redis有以下两种限制来避免出现这种情况，并且不同类型的客户端可以采取不同的配置。

  1. 大小限制:若客户端缓冲区超过某一个设定的值，直接关闭客户端连接。

  2. 持续性限制:若客户端缓冲区连续一定时间内一直超过某个设定的值时，则关闭客户端连接。

    client-output-buffer-limit normal 0 0 0
client-output-buffer-limit slave 256mb 64mb 60
client-output-buffer-limit pubsub 8mb 2mb 60

redis把客户端分为3种，一种是普通的客户端，比如应用程序获取的jedis连接、普通的redis-cli，上面配置的三个0意思是对于客户端缓冲区无限制；第二种slave客户端，上面配置的3个参数的意思是若缓冲区超过256m则redis服务器直接关闭客户端连接，或者持续60s缓冲区一直超过64m也会关闭客户端连接。第三种是pub/sub客户端配置。通过以上几种不同的配置，redis就可以避免客户端缓冲区过大导致占用过大内存引起的问题。

timeout

redis为了避免客户端连接数过多，有一个timeout配置，意思是如果连接的空闲时间超过了timeout的值，则关闭连接。默认配置是0，意思是没有超时限制，永远不关闭连接。这个timeout参数配置，要跟客户端创建的连接池的参数配合起来一起使用。即客户端检测线程的运行时间间隔必须小于timeout参数，才能避免redis主动断开连接。

检查Redis配置

Redis搭建使用Redis为主从哨兵模式

检查所有主从实例的timeout参数发现其中一个被设置为了6s，其小于客户端默认的30s，但是其他实例timeout参数均为0。所以当客户端被分配连接到该配置错误实例时会被断开连接，但连接到其他实例并不会。

重新设置该实例timeout为0，问题最终解决。

生产上timeout参数推荐设置为开启，此时需要注意与客户端配置匹配。

总结

  • Redis客户端与服务端均有自动断开连接的相关设置。

客户端使用如下参数控制校验连接：

# 获取连接时校验有效性
spring.redis.pool.testOnBorrow=true
# 创建连接时校验有效性
spring.redis.pool.testOnCreate=true
# 返还连接到连接池时校验有效性
spring.redis.pool.testOnReturn=true
# 连接空闲时定时校验有效性
spring.redis.pool.testWhileIdle=true
# 连接空闲时定时校验有效性间隔时间
spring.redis.pool.timeBetweenEvictionRunsMillis=30000

​ 服务端使用如下参数控制主动断开连接：

# 客户端输出缓冲区buffer大小限制
client-output-buffer-limit
# 空闲连接超时断开时间
timeout
  • spring-boot-starter-data-redis并不支持以上客户端的自动配置，需要自己手动配置连接工厂的连接池配置。

参考文章

1. JedisPool连接池相关配置

FEATURED TAGS