Question

我正在开发一个消耗大量数据集的spring boot命令行应用程序。数据集很麻烦，所以我想将它与我的代码分开，但我不想让用户负担必须管理数据。

我认为理想的解决方案是创建一个数据jar并从中读取主应用程序。不幸的是，当它在依赖jar中时，我无法成功读取数据。

我已经制作了一个示例应用来展示我到目前为止所做的尝试

https://github.com/LewisWatson/java-data-jar

. ├── data-jar │ ├── pom.xml │ └── src │ └── main │ └── resources │ └── data │ ├── hellos │ │ └── hello.txt │ └── loremIpsums │ └── loremIpsum.txt ├── data-jar-reader │ ├── pom.xml │ └── src │ └── main │ ├── java │ │ └── com │ │ └── example │ │ └── datajarreader │ │ └── DataJarReaderApplication.java │ └── resources │ └── application.properties ├── LICENSE ├── pom.xml └── README.md

它有两个模块：

data-jar在maven标准资源目录的data目录下的两个目录中包含两个文本文件。
data-jar-reader是Spring引导命令行应用程序，它依赖于data-jar并尝试访问其数据文件。

进行阅读的班级是DataJarReaderApplication

@SpringBootApplication
public class DataJarReaderApplication implements CommandLineRunner {

  private static final Logger log = LoggerFactory.getLogger(DataJarReaderApplication.class);

  FileSystem fileSystem;

  public static void main(String[] args) {
    SpringApplication.run(DataJarReaderApplication.class, args);
  }

  @Override
  public void run(String... args) throws Exception {
    Path path = getTestDataPath();
    List<String> data = loadTestData(path);
    log.info("data {}", data);
  }

  private Path getTestDataPath() throws URISyntaxException, IOException {

    Path path;

    URI uri = this.getClass().getResource("/data").toURI();
    log.info("uri: {}", uri);

    if (uri.getScheme().equals("jar")) {
      fileSystem = FileSystems.newFileSystem(uri, Collections.<String, Object>emptyMap());
      log.info("fileSystem: {}", fileSystem);
      path = fileSystem.getPath("/data");
    } else { // not in a jar, probably running in iDE
      path = Paths.get(uri);
    }

    return path;
  }

  private List<String> loadTestData(Path testDataDirectory) throws IOException {
    try (Stream<Path> path = Files.walk(testDataDirectory).sorted()) {
      return loadTrackData(path);
    }
  }

  private List<String> loadTrackData(Stream<Path> walk) throws IOException {

    List<String> data = new ArrayList<>();

    for (Iterator<Path> it = walk.iterator(); it.hasNext();) {

      Path path = it.next();

      if (path.getFileName().toString().endsWith(".txt")) {

        log.info("text file path: {}", path);

        List<String> dataFromFile = getData(path);
        data.addAll(dataFromFile);

      } else {

        log.info("non text file path: {}", path);

      }

    }

    return data;
  }

  private List<String> getData(Path path) throws IOException {

    List<String> data = new ArrayList<>();

    try (BufferedReader reader = Files.newBufferedReader(path, StandardCharsets.UTF_8)) {

      String line;

      while ((line = reader.readLine()) != null) {
        data.add(line);
      }

    }
    return data;
  }
}

运行应用程序时得到的输出显示它在数据jar中找到数据

jar:file:/home/lewis/workspace/java-data-jar/data-jar-reader/target/data-jar-reader-0.0.1-SNAPSHOT.jar!/BOOT-INF/lib/data-jar-1.0-SNAPSHOT.jar!/data

但是当它试图访问数据时，我得到一个没有这样的文件异常......

$ java -jar data-jar-reader/target/data-jar-reader-0.0.1-SNAPSHOT.jar 

  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::        (v1.5.7.RELEASE)

2017-10-16 09:35:46.238  INFO 18756 --- [           main] c.e.d.DataJarReaderApplication           : Starting DataJarReaderApplication v0.0.1-SNAPSHOT on mir with PID 18756 (/home/lewis/workspace/java-data-jar/data-jar-reader/target/data-jar-reader-0.0.1-SNAPSHOT.jar started by lewis in /home/lewis/workspace/java-data-jar)
2017-10-16 09:35:46.249  INFO 18756 --- [           main] c.e.d.DataJarReaderApplication           : No active profile set, falling back to default profiles: default
2017-10-16 09:35:46.376  INFO 18756 --- [           main] s.c.a.AnnotationConfigApplicationContext : Refreshing org.springframework.context.annotation.AnnotationConfigApplicationContext@443b7951: startup date [Mon Oct 16 09:35:46 BST 2017]; root of context hierarchy
2017-10-16 09:35:47.049  INFO 18756 --- [           main] o.s.j.e.a.AnnotationMBeanExporter        : Registering beans for JMX exposure on startup
2017-10-16 09:35:47.056  INFO 18756 --- [           main] c.e.d.DataJarReaderApplication           : uri: jar:file:/home/lewis/workspace/java-data-jar/data-jar-reader/target/data-jar-reader-0.0.1-SNAPSHOT.jar!/BOOT-INF/lib/data-jar-1.0-SNAPSHOT.jar!/data
2017-10-16 09:35:47.094  INFO 18756 --- [           main] c.e.d.DataJarReaderApplication           : fileSystem: /home/lewis/workspace/java-data-jar/data-jar-reader/target/data-jar-reader-0.0.1-SNAPSHOT.jar
2017-10-16 09:35:47.098  INFO 18756 --- [           main] utoConfigurationReportLoggingInitializer : 

Error starting ApplicationContext. To display the auto-configuration report re-run your application with 'debug' enabled.
2017-10-16 09:35:47.111 ERROR 18756 --- [           main] o.s.boot.SpringApplication               : Application startup failed

java.lang.IllegalStateException: Failed to execute CommandLineRunner
    at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:735) [spring-boot-1.5.7.RELEASE.jar!/:1.5.7.RELEASE]
    at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:716) [spring-boot-1.5.7.RELEASE.jar!/:1.5.7.RELEASE]
    at org.springframework.boot.SpringApplication.afterRefresh(SpringApplication.java:703) [spring-boot-1.5.7.RELEASE.jar!/:1.5.7.RELEASE]
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:304) [spring-boot-1.5.7.RELEASE.jar!/:1.5.7.RELEASE]
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1118) [spring-boot-1.5.7.RELEASE.jar!/:1.5.7.RELEASE]
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1107) [spring-boot-1.5.7.RELEASE.jar!/:1.5.7.RELEASE]
    at com.example.datajarreader.DataJarReaderApplication.main(DataJarReaderApplication.java:34) [classes!/:0.0.1-SNAPSHOT]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_131]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_131]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_131]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_131]
    at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48) [data-jar-reader-0.0.1-SNAPSHOT.jar:0.0.1-SNAPSHOT]
    at org.springframework.boot.loader.Launcher.launch(Launcher.java:87) [data-jar-reader-0.0.1-SNAPSHOT.jar:0.0.1-SNAPSHOT]
    at org.springframework.boot.loader.Launcher.launch(Launcher.java:50) [data-jar-reader-0.0.1-SNAPSHOT.jar:0.0.1-SNAPSHOT]
    at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:51) [data-jar-reader-0.0.1-SNAPSHOT.jar:0.0.1-SNAPSHOT]
Caused by: java.nio.file.NoSuchFileException: /data
    at com.sun.nio.zipfs.ZipPath.getAttributes(ZipPath.java:666) ~[zipfs.jar:1.8.0_131]
    at com.sun.nio.zipfs.ZipFileSystemProvider.readAttributes(ZipFileSystemProvider.java:294) ~[zipfs.jar:1.8.0_131]
    at java.nio.file.Files.readAttributes(Files.java:1737) ~[na:1.8.0_131]
    at java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:219) ~[na:1.8.0_131]
    at java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:276) ~[na:1.8.0_131]
    at java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:322) ~[na:1.8.0_131]
    at java.nio.file.FileTreeIterator.<init>(FileTreeIterator.java:72) ~[na:1.8.0_131]
    at java.nio.file.Files.walk(Files.java:3574) ~[na:1.8.0_131]
    at java.nio.file.Files.walk(Files.java:3625) ~[na:1.8.0_131]
    at com.example.datajarreader.DataJarReaderApplication.loadTestData(DataJarReaderApplication.java:63) [classes!/:0.0.1-SNAPSHOT]
    at com.example.datajarreader.DataJarReaderApplication.run(DataJarReaderApplication.java:40) [classes!/:0.0.1-SNAPSHOT]
    at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:732) [spring-boot-1.5.7.RELEASE.jar!/:1.5.7.RELEASE]
    ... 14 common frames omitted

Answer 1

如果您在IDE中运行应用程序，则不必以不同方式访问数据。这些数据文件始终是资源，例如类路径中的文件应该作为资源加载。

Spring有一个方便的资源加载器，可以帮助发现类路径中的数据文件：PathMatchingResourcePatternResolver。

例如，要获取data包中的所有.txt文件：

ClassLoader cl = this.getClass().getClassLoader(); 
ResourcePatternResolver resolver = new PathMatchingResourcePatternResolver(cl);
Resource[] resources = resolver.getResources("classpath:/data/**/*.txt") ;
for (Resource resource: resources){
    logger.info(resource.getFilename());
}

然后，您可以在每个资源上使用getInputStream来读取您的文件。

Answer 2

假设你真的想要使用jar，你有几个选择：

您可以在类路径中列出数据jar并使用ClassLoader.getResourceAsStream（relativePathWithinJar）。
您可以将数据jar放入某个目录，并使用Java API将其作为常规ZIP / JAR存档打开。

在第二种情况下，看一下JarFile类。要打开它，您必须提供jar文件的路径。您可以将其放入某个硬编码目录，放入当前目录或通过命令行参数，系统属性或类似的东西传递一些任意路径。打开它之后，您可以通过相对（in-jar）路径将单个文件作为JarEntries获取。

与任何大型文件一样，最好避免使用将整个数据加载到内存中的方法，并坚持使用InputStream进行按顺序读取的方法。

从依赖jar中扫描和读取文件

2 个答案: