Question

目前我有一个启用了docker容器的mesos集群。我使用mesos docker支持在我的集群上运行spark框架。我想从chronos安排一个火花提交工作。能不能让我知道步骤或最好的方法。我对mesos很新，我不知道chronos会如何找到提交应用程序的spark-submit命令。

Answer 1

您可以将您的应用程序与Spark发行版一起打包到Docker容器中，并在Chronos中创建一个Docker作业：

http://mesos.github.io/chronos/docs/api.html#adding-a-docker-job

例如，向Chronos REST API发送POST请求，如下所示：

Create PROCEDURE [dbo].[GetSubscriptionData]
AS
BEGIN
SET NOCOUNT ON;
WITH
[Sub_Parameters] AS
(
    SELECT  [SubscriptionID], [Parameters] = CONVERT(XML,a.[Parameters])
    FROM [Subscriptions] a
),
[MySubscriptions] AS
(
    SELECT DISTINCT [SubscriptionID], [ParameterName] = QUOTENAME(p.value('(Name)[1]', 'nvarchar(max)')),   [ParameterValue] = p.value('(Value)[1]', 'nvarchar(max)')
    FROM [Sub_Parameters] a
        CROSS APPLY [Parameters].nodes('/ParameterValues/ParameterValue') t(p)
),
[SubscriptionsAnalysis] AS
(
    SELECT  a.[SubscriptionID], a.[ParameterName],  [ParameterValue] =  
        (
            SELECT  STUFF((SELECT [ParameterValue] + ', ' as [text()] 
            FROM [MySubscriptions]  
            WHERE   [SubscriptionID] = a.[SubscriptionID]   AND [ParameterName] = a.[ParameterName] 
                FOR XML PATH('')    ),1, 0, '') +''
        )
    FROM [MySubscriptions] a
    GROUP BY a.[SubscriptionID],a.[ParameterName]
)

SELECT
DISTINCT (a.[SubscriptionID]),
c.[UserName] AS Owner, 
b.Name as ReportName,
Convert(XML,a.[ExtensionSettings]).value('(//ParameterValue/Value[../Name="RENDER_FORMAT"])[1]','nvarchar(50)') as ReportExtension,
b.Path,
a.[Locale], 
a.[InactiveFlags], 
d.[UserName] AS Modified_by, 
a.[ModifiedDate], 
a.[Description], 
a.[LastStatus], 
a.[EventType], 
a.[LastRunTime], 
a.[DeliveryExtension],
a.[Version],
sch.StartDate,
--e.[ParameterName],
--LEFT(e.[ParameterValue],LEN(e.[ParameterValue])-1) as [ParameterValue],
SUBSTRING(b.PATH,2,LEN(b.PATH)-(CHARINDEX('/',REVERSE(b.PATH))+1)) AS ProjectName
FROM 
    [Subscriptions] a 
    INNER JOIN [Catalog] AS b ON a.[Report_OID] = b.[ItemID]
    Inner Join ReportSchedule as RS on rs.SubscriptionID = a.SubscriptionID
    INNER JOIN Schedule AS Sch ON Sch.ScheduleID = rs.ScheduleID
    LEFT OUTER JOIN [Users] AS c ON a.[OwnerID] = c.[UserID]
    LEFT OUTER JOIN [Users] AS d ON a.MODIFIEDBYID = d.Userid
    LEFT OUTER JOIN [SubscriptionsAnalysis] AS e ON a.SubscriptionID = e.SubscriptionID;
 END

在chronos中运行spark-submit

1 个答案: