V2EX = way to explore
V2EX 是一个关于分享和探索的地方
现在注册
已注册用户请  登录
qwefdrt
V2EX  ?  Kafka

Kafka 内存陡然降低问题

  •  
  •   qwefdrt · 2019-01-09 10:13:54 +08:00 · 5525 次点击
    这是一个创建于 1906 天前的主题,其中的信息可能已经有所发展或是发生改变。
    最近一直出现 Kafka 集群中所有节点内存瞬间降低的问题。
    然后当时的日志里显示:
    [2019-01-09 01:08:05,297] WARN Send worker leaving thread (org.apache.zookeeper.server.quorum.QuorumCnxManager)
    [2019-01-09 01:08:05,296] INFO Received connection request /(org.apache.zookeeper.server.quorum.QuorumCnxManager)
    [2019-01-09 01:08:05,447] ERROR Unreasonable buffer length: 100000 (org.apache.zookeeper.server.quorum.QuorumCnxManager)
    [2019-01-09 01:08:05,547] INFO Received connection request /(org.apache.zookeeper.server.quorum.QuorumCnxManager)
    [2019-01-09 01:08:05,564] WARN Connection broken for id 8444275087900673, my id = 1, error = (org.apache.zookeeper.server.quorum.QuorumCnxManager)
    java.io.IOException: Received packet with invalid packet: 0
    at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1012)
    [2019-01-09 01:08:05,565] WARN Interrupting SendWorker (org.apache.zookeeper.server.quorum.QuorumCnxManager)
    [2019-01-09 01:08:05,595] WARN Interrupted while waiting for message on queue (org.apache.zookeeper.server.quorum.QuorumCnxManager)
    java.lang.InterruptedException
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
    at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
    at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1094)
    at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:74)
    at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:929)
    [2019-01-09 01:08:05,597] WARN Send worker leaving thread (org.apache.zookeeper.server.quorum.QuorumCnxManager)
    [2019-01-09 01:08:05,597] INFO Received connection request (org.apache.zookeeper.server.quorum.QuorumCnxManager)
    [2019-01-09 01:08:05,646] WARN Connection broken for id 3377699988963328, my id = 1, error = (org.apache.zookeeper.server.quorum.QuorumCnxManager)
    java.io.IOException: Received packet with invalid packet: 0
    at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1012)
    [2019-01-09 01:08:05,670] WARN Interrupting SendWorker (org.apache.zookeeper.server.quorum.QuorumCnxManager)
    [2019-01-09 01:08:05,670] WARN Interrupted while waiting for message on queue (org.apache.zookeeper.server.quorum.QuorumCnxManager)

    请问 ERROR Unreasonable buffer length: 100000 这个是什么原因造成的,应该如何解决?
    11 条回复  ?  2019-01-09 21:38:09 +08:00
    leonme
        1
    leonme  
       2019-01-09 10:17:24 +08:00
    问大家一个问题:
    leonme
        2
    leonme  
       2019-01-09 10:17:50 +08:00
    @leonme 忽略
    petelin
        3
    petelin  
       2019-01-09 11:27:44 +08:00
    我搜了一下网上的资料, 有一个贴了相关的源码, 看起来是要发送端发送的数据太大了(你这个例子是 10 万字节), 超过了缓冲区, 然后就 close socket 了. (一条消息不会这么大吧, 感觉也有可能是协议解析有问题, 读到是 10 万字节,其实没这么多)
    试试发送小一点的数据?

    期待下文.
    qwefdrt
        4
    qwefdrt  
    OP
       2019-01-09 11:41:10 +08:00
    @petelin 我也是这样认为的,不知道为什么会有这么大的数据。 请问有什么方法限制吗?
    petelin
        5
    petelin  
       2019-01-09 11:45:50 +08:00 via iPhone
    @qwefdrt 客户端业务逻辑限制一下?另外内存陡然下降是因为释放了 socket 吗?报错之后还能不能发消息接收消息
    qwefdrt
        6
    qwefdrt  
    OP
       2019-01-09 12:25:48 +08:00 via iPhone
    @petelin 目前还能收发消息,我一遇到这个问题就通过重启来解决哈哈
    hackerwin7
        7
    hackerwin7  
       2019-01-09 13:17:20 +08:00 via iPhone
    kafka 版本?
    qwefdrt
        8
    qwefdrt  
    OP
       2019-01-09 13:44:43 +08:00
    @hackerwin7 kafka_2.11-2.0.0
    hackerwin7
        9
    hackerwin7  
       2019-01-09 14:18:12 +08:00 via iPhone
    Kafka 日志? 写 zookeeper 写太大了
    qwefdrt
        10
    qwefdrt  
    OP
       2019-01-09 18:04:14 +08:00
    @hackerwin7 我贴的是 Kafka 日志。Zookeeper 写太大了具体指的是?
    hackerwin7
        11
    hackerwin7  
       2019-01-09 21:38:09 +08:00 via iPhone
    @qwefdrt 这是 zk 的日志,你是不是起的内嵌的 zk ? 然后我没有看到 broker 相关的日志所以才叫你贴 Kafka 的日志,然后这个 error unreasonable buffer length 其实是 zk 抛出来的,zk 在 parse 的时候一旦 datainpustream 的 length 小于 0 或者超过 Max 就会报这个
    关于   ·   帮助文档   ·   博客   ·   API   ·   FAQ   ·   我们的愿景   ·   实用小工具   ·   4644 人在线   最高记录 6543   ·     Select Language
    创意工作者们的社区
    World is powered by solitude
    VERSION: 3.9.8.5 · 26ms · UTC 09:50 · PVG 17:50 · LAX 02:50 · JFK 05:50
    Developed with CodeLauncher
    ? Do have faith in what you're doing.


    http://www.vxiaotou.com