When using the decode_cef processor to parse a CEF log, it fails to disregard empty fields of type IP or integer, resulting in the following error.
"error":{"message":["error in field 'cn1': strconv.ParseInt: parsing \"\": invalid syntax","error in field 'sourceTranslatedAddress': value is not a valid IP address","error in field 'dst': value is not a valid IP address","error in field 'destinationTranslatedAddress': value is not a valid IP address"]}
Sample CEF Log:
2536 <14>1 2024-07-04T09:16:35.992Z logfwd20-251f92c6-abd9-4da9-a32f-ea60baed66ca-taskmanager-wx85p logforwarder - panwlogs - CEF:0|Palo Alto Networks|LF|2.0|TRAFFIC|end|3|dtz=UTC rt=Jul 04 2024 09:03:48 deviceExternalId=no-serial PanOSConfigVersion=10.2 start=Jul 04 2024 09:03:39 src=8.211.42.91 dst= sourceTranslatedAddress= destinationTranslatedAddress= cs1=intrazone-default cs1Label=Rule suser= duser= app=unknown-udp cs3=vsys1 cs3Label=VirtualLocation cs4=untrust cs4Label=FromZone cs5=untrust cs5Label=ToZone deviceInboundInterface=ethernet1/1 deviceOutboundInterface=ethernet1/1 cs6=Cortex Data Lake cs6Label=LogSetting cn1= cn1Label=SessionID cnt=1 spt=13442 dpt=500 sourceTranslatedPort=0 destinationTranslatedPort=0 proto=udp act=allow PanOSBytes=82 out=82 in=0 cn2=1 cn2Label=PacketsTotal PanOSSessionStartTime=Jul 04 2024 09:03:05 cn3=0 cn3Label=SessionDuration cs2=any cs2Label=URLCategory externalId=7361339208201408573 PanOSSourceLocation=DE PanOSDestinationLocation=US PanOSPacketsSent=1 PanOSPacketsReceived=0 reason=aged-out PanOSDGHierarchyLevel1=65544 PanOSDGHierarchyLevel2=65545 PanOSDGHierarchyLevel3=65546 PanOSDGHierarchyLevel4=65550 PanOSVirtualSystemName= dvchost=GP cloud service cat=from-policy PanOSSourceUUID= PanOSDestinationUUID= PanOSIMSI=0 PanOSIMEI= PanOSParentSessionID=0 PanOSParentStarttime=Jan 01 1970 00:00:00 PanOSTunnel=N/A PanOSEndpointAssociationID=0 PanOSChunksTotal=0 PanOSChunksSent=0 PanOSChunksReceived=0 PanOSRuleUUID=40b8842f-eec3-4e04-b760-6a2ce4698fde PanOSHTTP2Connection=0 PanOSLinkChangeCount=0 PanOSSDWANPolicyName= PanOSLinkSwitches= PanOSSDWANCluster= PanOSSDWANDeviceType= PanOSSDWANClusterType= PanOSSDWANSite= PanOSDynamicUserGroupName= PanOSX-Forwarded-ForIP= PanOSSourceDeviceCategory= PanOSSourceDeviceProfile= PanOSSourceDeviceModel= PanOSSourceDeviceVendor= PanOSSourceDeviceOSFamily= PanOSSourceDeviceOSVersion= PanOSSourceDeviceHost= PanOSSourceDeviceMac= PanOSDestinationDeviceCategory= PanOSDestinationDeviceProfile= PanOSDestinationDeviceModel= PanOSDestinationDeviceVendor= PanOSDestinationDeviceOSFamily= PanOSDestinationDeviceOSVersion= PanOSDestinationDeviceHost= PanOSDestinationDeviceMac= PanOSContainerID= PanOSContainerNameSpace= PanOSContainerName= PanOSSourceEDL= PanOSDestinationEDL= PanOSGPHostID= PanOSEndpointSerialNumber= PanOSSourceDynamicAddressGroup= PanOSDestinationDynamicAddressGroup= PanOSHASessionOwner= PanOSTimeGeneratedHighResolution=Jul 04 2024 09:03:39 PanOSNSSAINetworkSliceType= PanOSNSSAINetworkSliceDifferentiator=
Filebeat configuration:
filebeat.inputs:
- type: tcp
host: "127.0.0.1:9528"
processors:
- copy_fields:
fields:
- from: message
to: event.original
fail_on_error: true
ignore_missing: false
- replace:
fields:
- field: "message"
pattern: "PanOSX-Forwarded-ForIP="
replacement: "PanOSXForwardedForIP="
- field: "message"
pattern: "PanOSX-Forwarded-For="
replacement: "PanOSXForwardedFor="
- field: "message"
pattern: "PanOSSplit-tunnelconfiguration="
replacement: "PanOSSplitTunnelconfiguration="
ignore_missing: false
fail_on_error: true
- decode_cef:
field: message
ecs: false
- drop_fields:
fields: ["message"]
ignore_missing: false
When using the decode_cef processor to parse a CEF log, it fails to disregard empty fields of type IP or integer, resulting in the following error.
"error":{"message":["error in field 'cn1': strconv.ParseInt: parsing \"\": invalid syntax","error in field 'sourceTranslatedAddress': value is not a valid IP address","error in field 'dst': value is not a valid IP address","error in field 'destinationTranslatedAddress': value is not a valid IP address"]}Sample CEF Log:
Filebeat configuration: