Skip to content

数据写入不到HDFS #5

@zhihui-ge

Description

@zhihui-ge

能留个钉钉或者微信什么的吗,方便交流
1.自定义了HttpSource,数据是可以读入的
2.数据写入不到HDFS里面去
======读入的数据=============75702552,DT35,1,230,2018-07-04 09:56:55,0,0,d9045b79b10df4f0307c158619d3c139,U093,0,0,0,953,3,0,10,0,0,12,71,1,0,0,0,0,0,0,0,0,2447740,0,0,2,7,23,0,0,0,0,1,0,0,0,0,sti9v7jz7OHQ3PbO0f4=,514,0,1,0,4,1,100,0,0,

18:59:30.562 [Source: httpreader (1/1)] INFO com.dtstack.flinkx.http.reader.HttpInputFormat - subtask input close finished
18:59:30.563 [flink-akka.actor.default-dispatcher-4] DEBUG org.apache.flink.api.common.io.DefaultInputSplitAssigner - No more input splits available
18:59:30.563 [flink-akka.actor.default-dispatcher-4] DEBUG org.apache.flink.runtime.jobmanager.JobManager - Send next input split null.
18:59:30.563 [Source: httpreader (1/1)] INFO com.dtstack.flinkx.http.reader.HttpInputFormat - subtask input close finished
18:59:30.565 [Source: httpreader (1/1)] DEBUG org.apache.flink.streaming.runtime.tasks.StreamTask - Finished task Source: httpreader (1/1)
18:59:30.565 [Source: httpreader (1/1)] INFO com.dtstack.flinkx.http.reader.HttpInputFormat - subtask input close finished
18:59:30.566 [Source: httpreader (1/1)] DEBUG org.apache.flink.streaming.runtime.tasks.StreamTask - Closed operators for task Source: httpreader (1/1)
18:59:30.577 [Source: httpreader (1/1)] DEBUG org.apache.flink.runtime.io.network.partition.PipelinedSubpartition - Finished PipelinedSubpartition [number of buffers: 5 (59570 bytes), finished? true, read view? false].
18:59:30.577 [Source: httpreader (1/1)] INFO org.apache.flink.runtime.taskmanager.Task - Source: httpreader (1/1) (16ac1fb0b1019a13e7bdb5d5826b1ba9) switched from RUNNING to FINISHED.
18:59:30.577 [Source: httpreader (1/1)] INFO org.apache.flink.runtime.taskmanager.Task - Freeing task resources for Source: httpreader (1/1) (16ac1fb0b1019a13e7bdb5d5826b1ba9).
18:59:30.578 [Source: httpreader (1/1)] DEBUG org.apache.flink.runtime.io.network.NetworkEnvironment - Unregister task Source: httpreader (1/1) from network environment (state: FINISHED).
18:59:30.578 [Source: httpreader (1/1)] INFO org.apache.flink.runtime.taskmanager.Task - Ensuring all FileSystem streams are closed for task Source: httpreader (1/1) (16ac1fb0b1019a13e7bdb5d5826b1ba9) [FINISHED]
18:59:30.579 [flink-akka.actor.default-dispatcher-4] INFO org.apache.flink.runtime.taskmanager.TaskManager - Un-registering task and sending final execution state FINISHED to JobManager for task Source: httpreader (16ac1fb0b1019a13e7bdb5d5826b1ba9)
18:59:30.598 [flink-akka.actor.default-dispatcher-2] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: httpreader (1/1) (16ac1fb0b1019a13e7bdb5d5826b1ba9) switched from RUNNING to FINISHED.
18:59:30.598 [flink-akka.actor.default-dispatcher-2] DEBUG org.apache.flink.runtime.instance.SlotSharingGroupAssignment - Release simple slot SimpleSlot (0)(0) - b66c222807be61af44639fa9dcdc8514 @ localhost (dataPort=-1) - RELEASED.
18:59:30.599 [flink-akka.actor.default-dispatcher-5] INFO org.apache.flink.runtime.client.JobSubmissionClientActor - 07/04/2018 18:59:30 Source: httpreader(1/1) switched to FINISHED
07/04/2018 18:59:30 Source: httpreader(1/1) switched to FINISHED
18:59:34.416 [flink-akka.actor.default-dispatcher-4] DEBUG org.apache.flink.runtime.taskmanager.TaskManager - Sending heartbeat to JobManager
18:59:39.416 [flink-akka.actor.default-dispatcher-2] DEBUG org.apache.flink.runtime.taskmanager.TaskManager - Sending heartbeat to JobManager
18:59:44.416 [flink-akka.actor.default-dispatcher-4] DEBUG org.apache.flink.runtime.taskmanager.TaskManager - Sending heartbeat to JobManager
18:59:46.586 [Sink: hdfswriter (1/1)] INFO com.dtstack.flinkx.hdfs.writer.HdfsTextOutputFormat - subtask[0] close()
18:59:49.416 [flink-akka.actor.default-dispatcher-4] DEBUG org.apache.flink.runtime.taskmanager.TaskManager - Sending heartbeat to JobManager
18:59:54.416 [flink-akka.actor.default-dispatcher-2] DEBUG org.apache.flink.runtime.taskmanager.TaskManager - Sending heartbeat to JobManager
18:59:59.416 [flink-akka.actor.default-dispatcher-2] DEBUG org.apache.flink.runtime.taskmanager.TaskManager - Sending heartbeat to JobManager
19:00:04.416 [flink-akka.actor.default-dispatcher-4] DEBUG org.apache.flink.runtime.taskmanager.TaskManager - Sending heartbeat to JobManager
19:00:09.415 [flink-akka.actor.default-dispatcher-4] DEBUG org.apache.flink.runtime.taskmanager.TaskManager - Sending heartbeat to JobManager
19:00:14.415 [flink-akka.actor.default-dispatcher-4] DEBUG org.apache.flink.runtime.taskmanager.TaskManager - Sending heartbeat to JobManager
19:00:19.415 [flink-akka.actor.default-dispatcher-4] DEBUG org.apache.flink.runtime.taskmanager.TaskManager - Sending heartbeat to JobManager

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions