我有一个rails应用程序,它部署在带有nginx和passenger的ec2实例上。对于后台工作,我使用了redki的sidekiq。我在服务器上有1 GB的内存。我还在nginx配置中为乘客设置了max_pool_size为1,并将sidekiq并发设置为1.
当乘客获得多个请求时,它会获取最大内存,而sidekiq因内存不足而被杀死。
Passenger和sidekiq在理想位置占用大约54%的内存,并且在完成此过程后没有释放内存。在理想位置的日志下面:
public class SparkDataSets {
private static SparkConf conf;
private static JavaSparkContext sc;
private static SQLContext sqlsc;
private static void run(String path) {
JavaRDD<Employee> rdd = sc.textFile(path).map((t) -> {
String[] values = t.split(",", -1);
Employee emp = new Employee();
SimpleDateFormat dateFormat = new SimpleDateFormat("yyyy-MM-yy");
Date startdate = dateFormat.parse(values[1]);
Date Enddate = dateFormat.parse(values[5]);
long start = startdate.getTime();
long end = Enddate.getTime();
emp.setId(Integer.parseInt(values[0]));
emp.setStart_Date(new Timestamp(start));
emp.setFirstName(values[2]);
emp.setLastName(values[3]);
emp.setGender(values[4]);
emp.setEndDate(new Timestamp(end));
emp.setDepartMent_id(values[6]);
return emp;
});
Dataset<Employee> set = sqlsc.createDataset(rdd.rdd(),Encoders.bean(Employee.class));
set.createOrReplaceTempView("Employee");
Dataset<Row> filterset = sqlsc.sql("select * from Employee where lastName='Facello'");
filterset.show();
Dataset<Employee> bean = sqlsc.createDataset(rdd.rdd(),Encoders.bean(Employee.class));
bean.show();
}
public static void main(String... args) {
conf = new SparkConf().setMaster("local").setAppName("GroupBy");
sc = new JavaSparkContext(conf);
sqlsc = new SQLContext(sc);
run("data/input/Employee.txt");
}
}
当乘客和sidekiq收到多个请求时,由于内存不足导致sidekiq被杀。
ps aux --sort=-%mem | awk 'NR<=10{print $0}'
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
ubuntu 13975 1.5 24.8 774064 251908 ? Sl 11:04 0:01 Passenger RubyApp: /var/www/dairylac/Quickmilk/current/public (staging)
ubuntu 14108 5.1 21.4 914332 217596 ? Sl 11:04 0:03 sidekiq 4.1.2 Quickmilk [0 of 1 busy]
ubuntu 13858 3.8 19.9 491072 202544 ? Sl 11:03 0:04 Passenger AppPreloader: /var/www/dairylac/Quickmilk/current
postgres 13986 0.2 4.4 192528 45048 ? Ss 11:04 0:00 postgres: enbake quickmilk_dev_one 127.0.0.1(32902) idle
postgres 1489 0.0 3.5 185752 36044 ? Ss Aug29 0:06 postgres: writer process
ubuntu 13952 0.0 1.9 42764 19376 ? Ssl 11:04 0:00 /var/www/dairylac/Quickmilk/shared/bundle/ruby/2.1.0/gems/appsignal-1.2.5/ext/appsignal-agent
root 13811 0.1 1.6 677924 16744 ? Sl 11:03 0:00 Passenger core
postgres 14117 0.0 1.4 187736 14316 ? Ss 11:04 0:00 postgres: enbake quickmilk_dev_one 127.0.0.1(32924) idle
postgres 13959 0.0 1.4 187736 14308 ? Ss 11:04 0:00 postgres: enbake quickmilk_dev_one 127.0.0.1(32900) idle
我没有找到为什么乘客和sidekiq被杀,我该怎么解决。
请给我任何建议。
提前致谢。