我有很多想要参加的Samza工作。我可以让第一个运行正常。但是,第二个作业似乎处于ACCEPTED状态,并且在我杀死第一份作业之前从未过渡到RUNNING状态。
以下是YARN UI的视图:
我有2个数据节点,所以我应该可以运行多个作业。这是我的public static void main(String[] args) {
Scanner input=null;
ArrayList<EmployeeFX> employeeList=new ArrayList<EmployeeFX>();
try {
FileReader Info=new FileReader("P01_DATA.txt");
input=new Scanner(Info).useDelimiter("\\s+"); //Single white space regex is enough.
}
catch(FileNotFoundException noFile) {
System.out.println("Can't open file");
System.exit(1);
}
input.nextLine(); // Ignore the first line
input.nextLine(); // Ignore the second line
try {
while(input.hasNext()) { //hasNext() will check for the next available token
employeeList.add(new EmployeeFX(input.nextInt(),input.next(),input.next(), input.nextBoolean(), input.nextInt()));
} // Additional newLine() reading is not required here.
}
catch(NoSuchElementException element) {
System.err.println("Wrong type of file");
System.exit(1);
}
catch(IllegalStateException state) {
System.err.println("Couldn't read from file");
System.exit(1);
}
if(input!=null) {
input.close();
}
}
的相关部分(我在文件中唯一的其他配置是与HA配置,Zookeeper等):
yarn-site.xml
编辑:
我可以在资源管理器日志中看到:
<property>
<name>yarn.scheduler.minimum-allocation-mb</name>
<value>128</value>
<description>Minimum limit of memory to allocate to each container request at the Resource Manager.</description>
</property>
<property>
<name>yarn.scheduler.maximum-allocation-mb</name>
<value>2048</value>
<description>Maximum limit of memory to allocate to each container request at the Resource Manager.</description>
</property>
<property>
<name>yarn.scheduler.minimum-allocation-vcores</name>
<value>1</value>
<description>The minimum allocation for every container request at the RM, in terms of virtual CPU cores. Requests lower than this won't take effect, and the specified value will get allocated the minimum.</description>
</property>
<property>
<name>yarn.scheduler.maximum-allocation-vcores</name>
<value>2</value>
<description>The maximum allocation for every container request at the RM, in terms of virtual CPU cores. Requests higher than this won't take effect, and will get capped to this value.</description>
</property>
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>4096</value>
<description>Physical memory, in MB, to be made available to running containers</description>
</property>
<property>
<name>yarn.nodemanager.resource.cpu-vcores</name>
<value>4</value>
<description>Number of CPU cores that can be allocated for containers.</description>
</property>
我不能正确做什么?
答案 0 :(得分:10)
答案在于资源管理器说没有足够的资源来创建新的samza容器和应用程序主机。
我将yarn.scheduler.capacity.maximum-am-resource-percent
中capacity-scheduler.xml
的值更改为默认值0.1。
此参数的documentation说明:
Maximum percent of resources in the cluster which can be used to run
application masters i.e. controls number of concurrent running applications.