Splunk log driver fails with index acknowledgement turned on #39986
Description
Description
I am trying to enable the log driver for Splunk, but it seems to be silently failing.
Steps to reproduce the issue:
Here is a bash script that reproduces the issue for me, with sensitive data replaced with env variables:
# This works just fine and shows up in Splunk for me
curl -k $URL/services/collector \
-H "Authorization: Splunk $TOKEN" \
-H "X-Splunk-Request-Channel: FE0ECFAD-13D5-401B-847D-77043BD77131" \
-d "{\"index\": \"$INDEX\", \"source\": \"$SOURCE\", \"event\":\"Showing that this works\"}"
docker run -d \
--publish 80:80 \
--log-driver=splunk \
--log-opt "splunk-token=$TOKEN" \
--log-opt "splunk-url=$URL" \
--log-opt "splunk-source=$SOURCE" \
--log-opt "splunk-insecureskipverify=true" \
--log-opt "splunk-index=$INDEX" \
nginx
# This event gets logged locally, but does not show up in Splunk
curl "localhost:80?please_show_up_in_splunk=true"
docker stop $(docker ps -q)Describe the results you received:
I am not seeing any logs being sent from my container to Splunk.
Describe the results you expected:
I expect to see logs sent from my container to Splunk.
Additional information you deem important (e.g. issue happens only occasionally):
I have seen others say that when index acknowledgement is turned on, the docker log driver silently fails. This does appear to be the case for our HEC, since the response from the initial curl is {"text":"Success","code":0,"ackId":1}, so it is indeed sending the acknowledgement.
UPDATE:
After looking at the code, I believe the problem is here. To use index acknowledgement, a header needs to be set (see here for more information, in the "About channels and sending data" section). Without this X-Splunk-Request-Channel header (which is a UUID), the POST request will fail.
Output of docker version:
Client: Docker Engine - Community
Version: 19.03.2
API version: 1.40
Go version: go1.12.8
Git commit: 6a30dfc
Built: Thu Aug 29 05:26:49 2019
OS/Arch: darwin/amd64
Experimental: false
Server: Docker Engine - Community
Engine:
Version: 19.03.2
API version: 1.40 (minimum version 1.12)
Go version: go1.12.8
Git commit: 6a30dfc
Built: Thu Aug 29 05:32:21 2019
OS/Arch: linux/amd64
Experimental: true
containerd:
Version: v1.2.6
GitCommit: 894b81a4b802e4eb2a91d1ce216b8817763c29fb
runc:
Version: 1.0.0-rc8
GitCommit: 425e105d5a03fabd737a126ad93d62a9eeede87f
docker-init:
Version: 0.18.0
GitCommit: fec3683
Output of docker info:
Client:
Debug Mode: false
Server:
Containers: 27
Running: 1
Paused: 0
Stopped: 26
Images: 672
Server Version: 19.03.2
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 894b81a4b802e4eb2a91d1ce216b8817763c29fb
runc version: 425e105d5a03fabd737a126ad93d62a9eeede87f
init version: fec3683
Security Options:
seccomp
Profile: default
Kernel Version: 4.9.184-linuxkit
Operating System: Docker Desktop
OSType: linux
Architecture: x86_64
CPUs: 6
Total Memory: 3.855GiB
Name: docker-desktop
ID: VA7U:7Z3N:IWDG:LECR:6IV5:IWOP:LQJD:TROR:SU4N:S2VH:5R37:EQLO
Docker Root Dir: /var/lib/docker
Debug Mode: true
File Descriptors: 35
Goroutines: 52
System Time: 2019-09-25T09:31:57.495390779Z
EventsListeners: 2
HTTP Proxy: gateway.docker.internal:3128
HTTPS Proxy: gateway.docker.internal:3129
Registry: https://index.docker.io/v1/
Labels:
Experimental: true
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine
Additional environment details (AWS, VirtualBox, physical, etc.):
This is being run locally right now, but I first found this issue when trying to use the log driver configuration on AWS Fargate, and it shows the same behavior there as well.