Skip to content

写入hdfs 失败 #987

@lordk911

Description

@lordk911

Describe the bug
使用1.12.4版本测试, 运行之前正常的任务,报错:

2022-06-22 15:15:49.849 [flink-akka.actor.default-dispatcher-3] INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcService  - Stopped Akka RPC service.
Exception in thread "main" org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
        at org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:144)
        at org.apache.flink.runtime.minicluster.MiniCluster.executeJobBlocking(MiniCluster.java:811)
        at com.dtstack.chunjun.environment.MyLocalStreamEnvironment.execute(MyLocalStreamEnvironment.java:174)
        at com.dtstack.chunjun.Main.exeSyncJob(Main.java:227)
        at com.dtstack.chunjun.Main.main(Main.java:122)
        at com.dtstack.chunjun.client.local.LocalClusterClientHelper.submit(LocalClusterClientHelper.java:35)
        at com.dtstack.chunjun.client.Launcher.main(Launcher.java:119)
Caused by: org.apache.flink.runtime.client.JobInitializationException: Could not instantiate JobManager.
        at org.apache.flink.runtime.dispatcher.Dispatcher.lambda$createJobManagerRunner$5(Dispatcher.java:488)
        at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.flink.runtime.client.JobExecutionException: Cannot initialize task 'Sink: hdfssinkfactory': null
        at org.apache.flink.runtime.executiongraph.ExecutionGraphBuilder.buildGraph(ExecutionGraphBuilder.java:293)
        at org.apache.flink.runtime.scheduler.SchedulerBase.createExecutionGraph(SchedulerBase.java:322)
        at org.apache.flink.runtime.scheduler.SchedulerBase.createAndRestoreExecutionGraph(SchedulerBase.java:276)
        at org.apache.flink.runtime.scheduler.SchedulerBase.<init>(SchedulerBase.java:249)
        at org.apache.flink.runtime.scheduler.DefaultScheduler.<init>(DefaultScheduler.java:133)
        at org.apache.flink.runtime.scheduler.DefaultSchedulerFactory.createInstance(DefaultSchedulerFactory.java:111)
        at org.apache.flink.runtime.jobmaster.JobMaster.createScheduler(JobMaster.java:342)
        at org.apache.flink.runtime.jobmaster.JobMaster.<init>(JobMaster.java:327)
        at org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.createJobMasterService(DefaultJobMasterServiceFactory.java:95)
        at org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.createJobMasterService(DefaultJobMasterServiceFactory.java:39)
        at org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.<init>(JobManagerRunnerImpl.java:163)
        at org.apache.flink.runtime.dispatcher.DefaultJobManagerRunnerFactory.createJobManagerRunner(DefaultJobManagerRunnerFactory.java:86)
        at org.apache.flink.runtime.dispatcher.Dispatcher.lambda$createJobManagerRunner$5(Dispatcher.java:472)
        ... 4 more
Caused by: java.lang.NullPointerException
        at com.dtstack.chunjun.connector.hdfs.sink.HdfsOrcOutputFormat.openSource(HdfsOrcOutputFormat.java:107)
        at com.dtstack.chunjun.sink.format.BaseFileOutputFormat.initVariableFields(BaseFileOutputFormat.java:112)
        at com.dtstack.chunjun.connector.hdfs.sink.BaseHdfsOutputFormat.initVariableFields(BaseHdfsOutputFormat.java:97)
        at com.dtstack.chunjun.sink.format.BaseFileOutputFormat.initializeGlobal(BaseFileOutputFormat.java:65)
        at org.apache.flink.runtime.jobgraph.InputOutputFormatVertex.initializeOnMaster(InputOutputFormatVertex.java:110)
        at org.apache.flink.runtime.executiongraph.ExecutionGraphBuilder.buildGraph(ExecutionGraphBuilder.java:289)
        ... 16 more

我把hdfs-site.xml\ core-site.xml 打入了chunjun-connector-hdfs-master.jar , 在之前的版本也是这么做的。

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions