虽然系统上有足够的空间,但Hudson失败并显示错误“设备上没有剩余空间”

时间:2010-07-26 20:30:17

标签: linux hudson

Hudson在构建了几个带有跟踪堆栈跟踪的项目后反复失败,错误“设备上没有剩余空间”,尽管磁盘上有足够的空间。任何文件夹都没有配额限制。下面是不同系统命令的输出。

以下是关键系统信息:

Hudson ver. 1.361
executable-war /opt/hudson/hudson.war
java.runtime.name OpenJDK Runtime Environment
java.runtime.version 1.6.0_18-b18
os.name Linux-Ubuntu 10.04
os.version 2.6.32-19-generic

根据df

,有50%的可用空间
$ df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda1            147550696  67382688  72672840  49% /
none                   1535580       304   1535276   1% /dev
none                   1539732       504   1539228   1% /dev/shm
none                   1539732        96   1539636   1% /var/run
none                   1539732         0   1539732   0% /var/lock
none                   1539732         0   1539732   0% /lib/init/rw
none                 147550696  67382688  72672840  49% /var/lib/ureadahead/debugfs

接下来我检查了lsof打开的文件句柄,那也是在限制

之内
lsof | wc -l
694

然后我通过这个命令检查文件句柄

cat /proc/sys/fs/file-nr
3392 0 306935

详细错误

[INFO] No tests to run.
[HUDSON] Recording test results
[INFO] [jar:jar {execution: default-jar}]
[INFO] Building jar: /opt/hudson/jobs/EIF_Branch_R1.2__Projects_PkgOnly/workspace/r1.2/projects/CRMServices/deployment/target/eif.deployment.CRMServices-1.2-bt.jar
[INFO] [antrun:run {execution: default}]
[INFO] Executing tasks
   [unzip] Expanding: /opt/hudson/jobs/EIF_Branch_R1.2__Projects_PkgOnly/workspace/r1.2/projects/CRMServices/deployment/src/it/resources/CRMServices.Driver-soapui-project.zip into /opt/hudson/jobs/EIF_Branch_R1.2__Projects_PkgOnly/workspace/r1.2/projects/CRMServices/deployment/src/it/resources
[HUDSON] Archiving /opt/hudson/jobs/EIF_Branch_R1.2__Projects_PkgOnly/workspace/r1.2/projects/CRMServices/deployment/pom.xml to /opt/hudson/jobs/EIF_Branch_R1.2__Projects_PkgOnly/modules/rogers.bt.deployment$eif.deployment.CRMServices/builds/2010-07-20_12-13-58/archive/rogers.bt.deployment/eif.deployment.CRMServices/1.2-bt/pom.xml
[HUDSON] Archiving /opt/hudson/jobs/EIF_Branch_R1.2__Projects_PkgOnly/workspace/r1.2/projects/CRMServices/deployment/target/eif.deployment.CRMServices-1.2-bt.jar to /opt/hudson/jobs/EIF_Branch_R1.2__Projects_PkgOnly/modules/rogers.bt.deployment$eif.deployment.CRMServices/builds/2010-07-20_12-13-58/archive/rogers.bt.deployment/eif.deployment.CRMServices/1.2-bt/eif.deployment.CRMServices-1.2-bt.jar
[HUDSON] Re-archiving /opt/hudson/jobs/EIF_Branch_R1.2__Projects_PkgOnly/workspace/r1.2/projects/CRMServices/deployment/target/eif.deployment.CRMServices-1.2-bt.jar
[INFO] ------------------------------------------------------------------------
[ERROR] FATAL ERROR
[INFO] ------------------------------------------------------------------------
[INFO] Failed to serialize hudson.model.Actionable#actions for class hudson.maven.MavenModuleSetBuild
No space left on device
[INFO] ------------------------------------------------------------------------
[INFO] Trace
java.langchannel stopped
ERROR: Failed to parse POMs
java.io.IOException: Remote call on Channel to Maven [/opt/bea/jdk160_05/bin/java, -cp, /opt/hudson/plugins/maven-plugin/WEB-INF/lib/maven-agent-1.363.jar:/opt/hudson/tools/Maven_2.2.1/boot/classworlds-1.1.jar, hudson.maven.agent.Main, /opt/hudson/tools/Maven_2.2.1, /opt/hudson/war/WEB-INF/lib/remoting-1.363.jar, /opt/hudson/plugins/maven-plugin/WEB-INF/lib/maven-interceptor-1.363.jar, 55951, /opt/hudson/plugins/maven-plugin/WEB-INF/lib/maven2.1-interceptor-1.2.jar] failed
                at hudson.remoting.Channel.call(Channel.java:564)
                at hudson.maven.ProcessCache$MavenProcess.call(ProcessCache.java:156)
                at hudson.maven.MavenModuleSetBuild$RunnerImpl.doRun(MavenModuleSetBuild.java:483)
                at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:416)
                at hudson.model.Run.run(Run.java:1253)
                at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:306)
                at hudson.model.ResourceController.execute(ResourceController.java:88)
                at hudson.model.Executor.run(Executor.java:127)
Caused by: java.lang.Error: Unable to load resource hudson/maven/Messages.properties
                at hudson.remoting.RemoteClassLoader.findResource(RemoteClassLoader.java:198)
                at java.lang.ClassLoader.getResource(ClassLoader.java:977)
                at java.lang.Class.getResource(Class.java:2074)
                at org.jvnet.localizer.ResourceBundleHolder.get(ResourceBundleHolder.java:83)
                at org.jvnet.localizer.ResourceBundleHolder.get(ResourceBundleHolder.java:102)
                at org.jvnet.localizer.ResourceBundleHolder.get(ResourceBundleHolder.java:102)
                at org.jvnet.localizer.ResourceBundleHolder.format(ResourceBundleHolder.java:139)
                at hudson.maven.Messages.MavenBuilder_AsyncFailed(Messages.java:233)
                at hudson.maven.MavenBuilder.call(MavenBuilder.java:184)
                at hudson.maven.MavenModuleSetBuild$Builder.call(MavenModuleSetBuild.java:696)
                at hudson.maven.MavenModuleSetBuild$Builder.call(MavenModuleSetBuild.java:640)
                at hudson.remoting.UserRequest.perform(UserRequest.java:114)
                at hudson.remoting.UserRequest.perform(UserRequest.java:48)
                at hudson.remoting.Request$2.run(Request.java:270)
                at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
                at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
                at java.util.concurrent.FutureTask.run(FutureTask.java:138)
                at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
                at java.lang.Thread.run(Thread.java:619)
Caused by: java.io.IOException: No space left on device
                at java.io.FileOutputStream.writeBytes(Native Method)
                at java.io.FileOutputStream.write(FileOutputStream.java:247)
                at hudson.remoting.RemoteClassLoader.makeResource(RemoteClassLoader.java:267)
                at hudson.remoting.RemoteClassLoader.findResource(RemoteClassLoader.java:194)
                ... 19 more
FATAL:  : No space left on device
hudson.util.IOException2:  : No space left on device
                at hudson.XmlFile.write(XmlFile.java:168)
                at hudson.model.Run.save(Run.java:1383)
                at hudson.maven.MavenModuleSetBuild$RunnerImpl.post2(MavenModuleSetBuild.java:595)
                at hudson.model.AbstractBuild$AbstractRunner.post(AbstractBuild.java:528)
                at hudson.model.Run.run(Run.java:1276)
                at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:306)
                at hudson.model.ResourceController.execute(ResourceController.java:88)
                at hudson.model.Executor.run(Executor.java:127)
Caused by: com.thoughtworks.xstream.io.StreamException:  : No space left on device
                at com.thoughtworks.xstream.core.util.QuickWriter.flush(QuickWriter.java:73)
                at com.thoughtworks.xstream.io.xml.PrettyPrintWriter.endNode(PrettyPrintWriter.java:288)
                at com.thoughtworks.xstream.io.WriterWrapper.endNode(WriterWrapper.java:37)
                at com.thoughtworks.xstream.io.path.PathTrackingWriter.endNode(PathTrackingWriter.java:48)
                at com.thoughtworks.xstream.core.TreeMarshaller.start(TreeMarshaller.java:99)
                at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.marshal(AbstractTreeMarshallingStrategy.java:38)
                at com.thoughtworks.xstream.XStream.marshal(XStream.java:840)
                at com.thoughtworks.xstream.XStream.marshal(XStream.java:829)
                at com.thoughtworks.xstream.XStream.toXML(XStream.java:804)
                at hudson.XmlFile.write(XmlFile.java:165)
                ... 7 more
Caused by: java.io.IOException: No space left on device
                at java.io.FileOutputStream.writeBytes(FileOutputStream.java)
                at java.io.FileOutputStream.write(FileOutputStream.java:260)
                at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:212)
                at java.io.BufferedWriter.flush(BufferedWriter.java:236)
                at hudson.util.AtomicFileWriter.flush(AtomicFileWriter.java:91)
                at com.thoughtworks.xstream.io.xml.PrettyPrintWriter.endNode(PrettyPrintWriter.java:288)
                at com.thoughtworks.xstream.io.path.PathTrackingWriter.endNode(PathTrackingWriter.java:49)
                at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.marshal(AbstractTreeMarshallingStrategy.java:38)
                at hudson.XmlFile.write(XmlFile.java:165)
                at hudson.model.Run.save(Run.java:1384)
                at hudson.maven.MavenModuleSetBuild$RunnerImpl.post2(MavenModuleSetBuild.java:595)
                at hudson.model.AbstractBuild$AbstractRunner.post(AbstractBuild.java:528) 

3 个答案:

答案 0 :(得分:26)

由于存储容量耗尽而不需要“设备上没有剩余空间”错误,因为它表明,它也可能是由于文件系统上的i节点耗尽造成的。换句话说,给定的文件系统只能包含这么多文件。运行df会表明一切都很好。

有关详细信息,请参阅this article

您需要删除一些您不再需要的文件,或者将Hudson放在不同的文件系统上。

在你知道你会有很多小文件构建一个带有明显更大的inode表的文件系统的情况下,这种情况并不少见。

答案 1 :(得分:1)

这是由于内存溢出造成的。

运行此命令以检查可用内存大小

df

如果您发现任何磁盘的100%或接近100%,请删除该目录中的数据,但请小心,然后重试。

在大多数情况下,/ tmp目录将是100%,因此重新启动计算机

sudo reboot

它将清除tmp目录中的所有文件。

答案 2 :(得分:0)

同时执行ipcs -uipcs -p,看看"空间不足" SHM内存已满,应用程序正在使用它。如果分配的等于sysctl kernel.shmmni,那么您将使用" full" SHM。您还可以与ipcs -l输出进行比较,因为这显示了系统限制。

最新版本的java可能会滥用shm用法,直到它没有空闲。关闭滥用shm的程序或增加kernel.shmmnikernel.shmmax sysctl以增加可用的shm空间