Question

我对Azure的块Blob存储有问题。问题是，如果我尝试访问存储并从罐子中创建一个容器，它可以正常工作。如果我尝试从spark-submit命令运行它，则它不起作用。我正在尝试捕获我的代码与Azure之间的流量，以查看出现问题的地方，但问题是尽管我可以在访问其他网站（如www.google.com）时捕获流量，但提琴手却无法捕获此类流量。 / p>

这有效：

import java.net.*;
import java.io.*;

public class Example
{
    public static void main(String[] args) throws Exception
    {
        System.setProperty("proxySet", "true");
        System.setProperty("proxyHost", "127.0.0.1");
        System.setProperty("proxyPort", "9090");
        System.setProperty("javax.net.ssl.trustStore", "C:\\data\\keys\\FiddlerKeystore");
        System.setProperty("javax.net.ssl.trustStorePassword", "password");


        URL x = new URL("https://www.google.com");
        HttpURLConnection hc = (HttpURLConnection)x.openConnection();

        hc.setRequestProperty("User-Agent","Mozilla/5.0 (Windows NT 6.0) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.121 Safari/535.2");

        InputStream is = hc.getInputStream();

        int u = 0;
        byte[] kj = new byte[1024];
        while((u = is.read(kj)) != -1)
        {
            System.out.write(kj,0,u);
        }
        is.close();
    }
}

Here's a screenshot of fiddler capturing Google traffic

现在，如果我对Azure代码执行相同操作，则Fiddler不会捕获任何内容：这是我的Azure代码：

import azure.AzureBlockBlobClient;
import common.AzureConf;
import org.apache.log4j.BasicConfigurator;

import java.io.IOException;


public class AzureExample {

    private AzureBlockBlobClient azureBlockBlobClient;
    private static final org.slf4j.Logger log = org.slf4j.LoggerFactory.getLogger(AzureExample.class);



    public AzureExample() {
        azureBlockBlobClient = new AzureBlockBlobClient(AzureConf.ACCOUNT_NAME,AzureConf.ACCOUNT_KEY, AzureConf.CONTAINER_NAME);
        azureBlockBlobClient.createContainer();
    }



    public static void main(String... args) throws IOException {
        BasicConfigurator.configure();
        System.setProperty("proxySet", "true");
        System.setProperty("proxyHost", "127.0.0.1");
        System.setProperty("proxyPort", "9090");
        System.setProperty("javax.net.ssl.trustStore", "C:\\data\\keys\\FiddlerKeystore");
        System.setProperty("javax.net.ssl.trustStorePassword", "password");

        new AzureExample();
        System.exit(0);
    }
}

这是连接到Azure的客户端：

    public AzureBlockBlobClient(String accountName, String accountKey, String containerName) {
    this.accountName = accountName;
    this.accountKey = accountKey;
    this.containerName = containerName;
    init();
}

private void init() {
    log.info("Init AzureBlockBlobClient started...");
    try {
        SharedKeyCredentials creds = new SharedKeyCredentials(accountName, accountKey);
        serviceURL = new ServiceURL(new URL("https://" + accountName + ".blob.core.windows.net/"),
            StorageURL.createPipeline(creds, new PipelineOptions()));
        containerURL = serviceURL.createContainerURL(containerName);
    }catch (InvalidKeyException e){
        log.error("Authentication error while trying to access storage account", e);
    }catch (MalformedURLException e) {
        log.error("Invalid Service URL", e);
        e.printStackTrace();
    }catch (Exception e) {
        e.printStackTrace();
        log.error("Error initializing AzureBlockBlobClient", e);
    }

    log.info("Init AzureBlockBlobClient Done!");
}

public void createContainer(){
    try {
        // Let's create a container using a blocking call to Azure Storage
        // If container exists, we'll catch and continue
        log.info("Creating container {}." , containerName);
        ContainerCreateResponse response = containerURL.create(null, null, null).blockingGet();
        log.info("Container Create Response was {}." , response.statusCode());
    }
    catch (RestException e){
        if (e instanceof RestException && e.response().statusCode() != 409) {
            log.error("Error Creating container", e);
        } else {
            log.info("Container {} already exists, resuming...", containerName);
        }
    }
}

这是我的常数所在的地方

public interface AzureConf {
String ACCOUNT_KEY ="<SomeAccountKey>";
String ACCOUNT_NAME = "storage";
String CONTAINER_NAME = "My-container";
}

这是我的maven pom.xml文件：

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
     xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
     xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>

<groupId>examples</groupId>
<artifactId>spark-azure-storage</artifactId>
<version>1.0-SNAPSHOT</version>
<properties>
    <junit.version>4.12</junit.version>
</properties>

<dependencies>
    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-azure</artifactId>
        <version>2.7.1</version>
        <scope>compile</scope>
    </dependency>
    <dependency>
        <groupId>com.microsoft.azure</groupId>
        <artifactId>azure-storage</artifactId>
        <version>2.0.0</version>
        <scope>runtime</scope>
    </dependency>
    <dependency>
        <groupId>com.microsoft.azure</groupId>
        <artifactId>azure-storage-blob</artifactId>
        <version>10.1.0</version>
    </dependency>
    <dependency>
        <groupId>junit</groupId>
        <artifactId>junit</artifactId>
        <version>${junit.version}</version>
        <scope>runtime</scope>
    </dependency>
    <dependency>
        <groupId>io.reactivex.rxjava2</groupId>
        <artifactId>rxjava</artifactId>
        <version>2.2.3</version>
    </dependency>
    <dependency>
        <groupId>org.slf4j</groupId>
        <artifactId>slf4j-api</artifactId>
        <version>1.7.16</version>
    </dependency>
    <!-- https://mvnrepository.com/artifact/com.typesafe.akka/akka-actor -->


    <dependency>
        <groupId>com.microsoft.rest.v2</groupId>
        <artifactId>client-runtime</artifactId>
        <version>2.0.0</version>
        <!--I have to exclude following dependencies and include version 2.9.7 of them otherwise I get
        SoSuchMethodError-->
        <exclusions>
            <exclusion>
                <groupId>com.fasterxml.jackson.core</groupId>
                <artifactId>jackson-annotations</artifactId>
            </exclusion>
            <exclusion>
                <groupId>com.fasterxml.jackson.core</groupId>
                <artifactId>jackson-core</artifactId>
            </exclusion>
            <exclusion>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-databind</artifactId>
            </exclusion>
        </exclusions>
    </dependency>
    <dependency>
        <groupId>org.slf4j</groupId>
        <artifactId>slf4j-log4j12</artifactId>
        <version>1.7.16</version>
    </dependency>
    <dependency>
        <groupId>com.fasterxml.jackson.core</groupId>
        <artifactId>jackson-core</artifactId>
        <version>2.9.7</version>
    </dependency>
    <dependency>
    <groupId>com.fasterxml.jackson.core</groupId>
    <artifactId>jackson-annotations</artifactId>
    <version>2.9.7</version>
    </dependency>
    <dependency>
        <groupId>com.fasterxml.jackson.core</groupId>
        <artifactId>jackson-databind</artifactId>
        <version>2.9.7</version>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-core_2.11</artifactId>
        <version>2.2.1</version>
    </dependency>
</dependencies>
<build>
    <plugins>
        <plugin>
            <groupId>org.codehaus.mojo</groupId>
            <artifactId>exec-maven-plugin</artifactId>
        </plugin>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-compiler-plugin</artifactId>
            <version>3.8.0</version>
            <configuration>
                <source>1.8</source>
                <target>1.8</target>
            </configuration>
        </plugin>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-shade-plugin</artifactId>
            <version>2.4.3</version>
            <executions>
                <execution>
                    <phase>package</phase>
                    <goals>
                        <goal>shade</goal>
                    </goals>
                </execution>
            </executions>
            <configuration>
                <filters>
                    <filter>
                        <artifact>*:*</artifact>
                        <excludes>
                            <exclude>META-INF/*.SF</exclude>
                            <exclude>META-INF/*.DSA</exclude>
                            <exclude>META-INF/*.RSA</exclude>
                        </excludes>
                    </filter>
                </filters>
                <transformers>
                    <transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
                        <resource>reference.conf</resource>
                    </transformer>
                    <transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
                    <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer"></transformer>
                </transformers>
            </configuration>
        </plugin>
    </plugins>
</build>

是否有帮助使其正常工作？预先谢谢你

Answer 1

根据Oracle官方文档Java Networking and Proxies，对于Java系统属性，没有这些属性proxySet，proxyHost和proxyPort。

请使用https.proxyHost和https.proxyPort代替它们，这对我有用。

使用提琴手捕获Azure块Blob通信

1 个答案: