2013年7月27日星期六

hadoop hdfs write too fast will a fault Connection reset by peer

server side error is as follows:
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcher.read0 (Native Method)
at sun.nio.ch.SocketDispatcher.read (SocketDispatcher.java: 21)
at sun.nio.ch.IOUtil.readIntoNativeBuffer (IOUtil.java: 198)
at sun.nio.ch.IOUtil.read (IOUtil.java: 171)
at sun.nio.ch.SocketChannelImpl.read (SocketChannelImpl.java: 243)
at org.apache.hadoop.ipc.Server.channelRead (Server.java: 1720)
at org.apache.hadoop.ipc.Server.access $ 2700 (Server.java: 94)
at org.apache.hadoop.ipc.Server $ Connection.readAndProcess (Server.java: 1094)
at org.apache.hadoop.ipc.Server $ Listener.doRead (Server.java: 537)
at org.apache.hadoop.ipc.Server $ Listener $ Reader.run (Server.java: 344)
at java.util.concurrent.ThreadPoolExecutor $ Worker.runTask (ThreadPoolExecutor.java: 886)
at java.util.concurrent.ThreadPoolExecutor $ Worker.run (ThreadPoolExecutor.java: 908)
at java.lang.Thread.run (Thread.java: 662)


client-side error is as follows:
2012-12-18 15:11:55 org.apache.hadoop.hdfs.DFSClient $ DFSOutputStream $ DataStreamer run
Warning: DataStreamer Exception: java.io.IOException: An existing connection was forcibly closed by the remote host
at sun.nio.ch.SocketDispatcher.write0 (Native Method)
at sun.nio.ch.SocketDispatcher.write (SocketDispatcher.java: 33)
at sun.nio.ch.IOUtil.writeFromNativeBuffer (IOUtil.java: 100)
at sun.nio.ch.IOUtil.write (IOUtil.java: 71)
at sun.nio.ch.SocketChannelImpl.write (SocketChannelImpl.java: 334)
at org.apache.hadoop.net.SocketOutputStream $ Writer.performIO (SocketOutputStream.java: 55)
at org.apache.hadoop.net.SocketIOWithTimeout.doIO (SocketIOWithTimeout.java: 142)
at org.apache.hadoop.net.SocketOutputStream.write (SocketOutputStream.java: 146)
at org.apache.hadoop.net.SocketOutputStream.write (SocketOutputStream.java: 107)
at java.io.BufferedOutputStream.write (BufferedOutputStream.java: 105)
at java.io.DataOutputStream.write (DataOutputStream.java: 90)
at org.apache.hadoop.hdfs.DFSClient $ DFSOutputStream $ DataStreamer.run (DFSClient.java: 2857)

write slowly seems to have no problem
------ For reference only --------------------------- ------------
personally think that this error can be ignored. .
general situation is caused by the environment, such as CPU fills, network delay. . . . . . . . . . .
------ For reference only -------------------------------------- -
found online about speaking and NIC driver, I tried to write linux would not have this problem. But I upgrade NIC driver Regardless
------ For reference only ------------------------------- --------
understand the whole, the entire bandwidth is used up, and then connect the broken estimated connection will regularly ping it, because no bandwidth, so that the other side is broken.

1 条评论:

  1. Thank you, my boss assigned me such a task and i could do it fast.Thank u very much

    回复删除