Question

我在本地计算机上使用DirectRunner尝试了SortValues转换的示例代码（Windows）

PCollection<KV<String, KV<String, Integer>>> input = ...

PCollection<KV<String, Iterable<KV<String, Integer>>>> grouped =
input.apply(GroupByKey.<String, KV<String, Integer>>create());

PCollection<KV<String, Iterable<KV<String, Integer>>>> groupedAndSorted =
grouped.apply(SortValues.<String, String, Integer>create(BufferedExternalSorter.options()));

但我收到错误PipelineExecutionException: java.lang.NoClassDefFoundError: org/apache/hadoop/io/Writable。这是否意味着此转换功能仅适用于Hadoop环境？

Answer 1

截至今天，如果您使用版本低于2.0.0的版本的Beam，则必须在maven pom文件中添加两个hadoop依赖项，以使此SortValues模块正常工作。

添加index.js:87 Error: EPERM: operation not permitted, open 'C:\Users\Noah Hefner\.atom\compile-cache\less\40e22529c8729b65e1d700a7ae3f73d57ed1288f\imports.json' at Error (native) at Object.fs.openSync (fs.js:640:18) at Object.module.(anonymous function) [as openSync] (ELECTRON_ASAR.js:168:20) at Object.fs.writeFileSync (fs.js:1343:33) at Proxy.writeFileSync (C:\Users\Noah Hefner\AppData\Local\atom\app-1.18.0\resources\app\node_modules\fs-plus\lib\fs-plus.js:298:29) at LessCache.module.exports.LessCache.writeJson (C:\Users\Noah Hefner\AppData\Local\atom\app-1.18.0\resources\app\node_modules\less-cache\lib\less-cache.js:212:23) at LessCache.module.exports.LessCache.setImportPaths (C:\Users\Noah Hefner\AppData\Local\atom\app-1.18.0\resources\app\node_modules\less-cache\lib\less-cache.js:161:18) at new LessCache (C:\Users\Noah Hefner\AppData\Local\atom\app-1.18.0\resources\app\node_modules\less-cache\lib\less-cache.js:68:18) at new LessCompileCache (C:\Users\Noah Hefner\AppData\Local\atom\app-1.18.0\resources\app\src\less-compile-cache.js:21:26) at ThemeManager.module.exports.ThemeManager.loadLessStylesheet (C:\Users\Noah Hefner\AppData\Local\atom\app-1.18.0\resources\app\src\theme-manager.js:283:32) at ThemeManager.module.exports.ThemeManager.loadStylesheet (C:\Users\Noah Hefner\AppData\Local\atom\app-1.18.0\resources\app\src\theme-manager.js:271:27) at ThemeManager.module.exports.ThemeManager.requireStylesheet (C:\Users\Noah Hefner\AppData\Local\atom\app-1.18.0\resources\app\src\theme-manager.js:191:30) at ThemeManager.module.exports.ThemeManager.reloadBaseStylesheets (C:\Users\Noah Hefner\AppData\Local\atom\app-1.18.0\resources\app\src\theme-manager.js:252:25) at ThemeManager.module.exports.ThemeManager.loadBaseStylesheets (C:\Users\Noah Hefner\AppData\Local\atom\app-1.18.0\resources\app\src\theme-manager.js:248:25) at AtomEnvironment.module.exports.AtomEnvironment.initialize (C:\Users\Noah Hefner\AppData\Local\atom\app-1.18.0\resources\app\src\atom-environment.js:339:25) at module.exports (C:\Users\Noah Hefner\AppData\Local\atom\app-1.18.0\resources\app\src\initialize-application-window.js:102:29) at setupWindow (file:///C:/Users/Noah%20Hefner/AppData/Local/atom/app-1.18.0/resources/app/static/index.js:106:12) at window.onload (file:///C:/Users/Noah%20Hefner/AppData/Local/atom/app-1.18.0/resources/app/static/index.js:67:9)handleSetupError @ index.js:87版本2.7.3或更高版本
添加hadoop-common版本2.7.3或更高版本。

否则，您只需要使用发布版本＆gt; = 2.0.0的Beam。

SortValues转换为仅在hadoop环境中运行的Beam中的Java SDK扩展吗？

1 个答案: