Tensorflow数据集上的分片操作是否具有确定性?

时间:2019-07-11 15:31:42

标签: python tensorflow

Tensorflow数据集具有Picked up _JAVA_OPTIONS: -Djava.net.preferIPv4Stack=true [INFO] Scanning for projects... [INFO] [INFO] -----------------< iav.main:C-Check.ParserDevelopment >----------------- [INFO] Building C-Check.ParserDevelopment 1.0-SNAPSHOT [INFO] --------------------------------[ jar ]--------------------------------- [INFO] [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ C-Check.ParserDevelopment --- [INFO] Deleting C:\Users\***\Desktop\C-Check.ParserDevelopment\target [INFO] [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ C-Check.ParserDevelopment --- [INFO] Using 'ISO-8859-1' encoding to copy filtered resources. [INFO] Copying 0 resource [INFO] [INFO] --- maven-compiler-plugin:3.3:compile (default-compile) @ C-Check.ParserDevelopment --- [INFO] Changes detected - recompiling the module! [INFO] Compiling 113 source files to C:\Users\***\Desktop\C-Check.ParserDevelopment\target\classes [INFO] /C:/Users/***/Desktop/C-Check.ParserDevelopment/src/main/java/iav/main/Args4J/Config.java: Some input files use or override a deprecated API. [INFO] /C:/Users/***/Desktop/C-Check.ParserDevelopment/src/main/java/iav/main/Args4J/Config.java: Recompile with -Xlint:deprecation for details. [INFO] ------------------------------------------------------------- [ERROR] COMPILATION ERROR : [INFO] ------------------------------------------------------------- [ERROR] /C:/Users/***/Desktop/C-Check.ParserDevelopment/src/main/java/iav/main/fileParser/LineNumberCalculatorTest.java:[10,19] incompatible types: boolean cannot be converted to org.eclipse.jdt.annotation.DefaultLocation [ERROR] /C:/Users/***/Desktop/C-Check.ParserDevelopment/src/main/java/iav/main/utils/StringUtilsTest.java:[10,19] incompatible types: boolean cannot be converted to org.eclipse.jdt.annotation.DefaultLocation [ERROR] /C:/Users/***/Desktop/C-Check.ParserDevelopment/src/main/java/iav/main/utils/package-info.java:[1,19] incompatible types: boolean cannot be converted to org.eclipse.jdt.annotation.DefaultLocation [ERROR] /C:/Users/***/Desktop/C-Check.ParserDevelopment/src/main/java/iav/main/Args4J/package-info.java:[7,19] incompatible types: boolean cannot be converted to org.eclipse.jdt.annotation.DefaultLocation [ERROR] /C:/Users/***/Desktop/C-Check.ParserDevelopment/src/main/java/iav/main/Args4J/spi/package-info.java:[5,19] incompatible types: boolean cannot be converted to org.eclipse.jdt.annotation.DefaultLocation [ERROR] /C:/Users/***/Desktop/C-Check.ParserDevelopment/src/main/java/iav/main/arguments/package-info.java:[1,19] incompatible types: boolean cannot be converted to org.eclipse.jdt.annotation.DefaultLocation [ERROR] /C:/Users/***/Desktop/C-Check.ParserDevelopment/src/main/java/iav/main/fileParser/ANTLR4/package-info.java:[1,19] incompatible types: boolean cannot be converted to org.eclipse.jdt.annotation.DefaultLocation [INFO] 7 errors [INFO] ------------------------------------------------------------- [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 16.023 s [INFO] Finished at: 2019-07-11T17:14:18+02:00 [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.3:compile (default-compile) on project C-Check.ParserDevelopment: Compilation failure: Compilation failure: [ERROR] /C:/Users/***/Desktop/C-Check.ParserDevelopment/src/main/java/iav/main/fileParser/LineNumberCalculatorTest.java:[10,19] incompatible types: boolean cannot be converted to org.eclipse.jdt.annotation.DefaultLocation [ERROR] /C:/Users/***/Desktop/C-Check.ParserDevelopment/src/main/java/iav/main/utils/StringUtilsTest.java:[10,19] incompatible types: boolean cannot be converted to org.eclipse.jdt.annotation.DefaultLocation [ERROR] /C:/Users/***/Desktop/C-Check.ParserDevelopment/src/main/java/iav/main/utils/package-info.java:[1,19] incompatible types: boolean cannot be converted to org.eclipse.jdt.annotation.DefaultLocation [ERROR] /C:/Users/***/Desktop/C-Check.ParserDevelopment/src/main/java/iav/main/Args4J/package-info.java:[7,19] incompatible types: boolean cannot be converted to org.eclipse.jdt.annotation.DefaultLocation [ERROR] /C:/Users/***/Desktop/C-Check.ParserDevelopment/src/main/java/iav/main/Args4J/spi/package-info.java:[5,19] incompatible types: boolean cannot be converted to org.eclipse.jdt.annotation.DefaultLocation [ERROR] /C:/Users/***/Desktop/C-Check.ParserDevelopment/src/main/java/iav/main/arguments/package-info.java:[1,19] incompatible types: boolean cannot be converted to org.eclipse.jdt.annotation.DefaultLocation [ERROR] /C:/Users/***/Desktop/C-Check.ParserDevelopment/src/main/java/iav/main/fileParser/ANTLR4/package-info.java:[1,19] incompatible types: boolean cannot be converted to org.eclipse.jdt.annotation.DefaultLocation [ERROR] -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException 操作,可创建给定数据集的唯一子集。

我们可以使用它来对数据集进行分区,如下所示:

Token: xxxx 

此分区确定吗? 即上面的3个子集将始终被赋予相同的元素?

documentation陈述有关shard的以下内容:

  

创建一个仅包含此数据集1 / num_shards的数据集。

     

此数据集运算符在运行分布式时非常有用   培训,因为它允许每个工人读取唯一的子集。

1 个答案:

答案 0 :(得分:0)

是的,绝对如此。这是确定性操作。

在上面的示例中,subset_one包含第一个元素,第五个元素等([0,4,8, ...]),而subset_two包含[1,5,9, ...],依此类推。