Skip to content

[Filebeat][httpjson]Ability to persist state between events #19071

@P1llus

Description

@P1llus

Creating an initial issue for discussion based on comment from @andrewkroh

When querying an API there is a need to persist state between each API call. Currently we do not have the possibility to store state in the registry file, so this issue is to track different user stories that would be great to support in the future.

User story:
Azure ATP Allows querying with URL parameters using ODATA queries documented:
https://docs.microsoft.com/en-us/windows/security/threat-protection/microsoft-defender-atp/exposed-apis-odata-samples

When starting up a beat for the first time, it will retrieve all events, and for each consequent event it would be nice to be able to track a value from the last ingested event to be used as a query filter.

Initial request:
https://api-eu.securitycenter.windows.com/api/alerts

Response:

{
	"@odata.context": "https://api-eu.securitycenter.windows.com/api/$metadata#Alerts",
	"value": [
		{
			"id": "da637249026486247330_1412635475",
			"incidentId": 2,
			"investigationId": null,
			"assignedTo": null,
			"severity": "Medium",
			"status": "New",
			"classification": null,
			"determination": null,
			"investigationState": "UnsupportedAlertType",
			"detectionSource": "WindowsDefenderAtp",
			"category": "InitialAccess",
			"threatFamilyName": null,
			"title": "Suspicious behavior by Microsoft Word was observed",
			"description": "A suspicious behavior by Microsoft Word application was observed. This behavior may indicate that a Word document was used to deliver malware or initiate other malicious activities on the machine.",
			"alertCreationTime": "2020-05-12T17:50:48.5153821Z",
			"firstEventTime": "2020-05-12T17:49:44.3584856Z",
			"lastEventTime": "2020-05-12T17:50:11.9889024Z",
			"lastUpdateTime": "2020-05-12T20:15:58.81Z",
			"resolvedTime": null,
			"machineId": "4d0ddab627ea0e54dc6ea75196b4b39db297f110",
			"computerDnsName": "testmachine1",
			"aadTenantId": "2978caa4-d66c-4c7e-9e30-40034eb7c6f3",
			"relatedUser": {
				"userName": "administrator1",
				"domainName": "TestMachine1"
			},
			"comments": [],
			"evidence": []
		},
		{
			"id": "da637249026573326737_-932580695",
			"incidentId": 1,
			"investigationId": null,
			"assignedTo": null,
			"severity": "Low",
			"status": "New",
			"classification": null,
			"determination": null,
			"investigationState": "InnerFailure",
			"detectionSource": "WindowsDefenderAv",
			"category": "Malware",
			"threatFamilyName": "Wintapp",
			"title": "'Wintapp' backdoor was detected",
			"description": "Backdoors are malicious remote access tools that allow attackers to access and control infected machines. Backdoors can also be used to exfiltrate data.\n\nThis detection might indicate that the malware was stopped from delivering its payload. However, it is prudent to check the machine for signs of infection.",
			"alertCreationTime": "2020-05-12T17:50:57.3326737Z",
			"firstEventTime": "2020-05-12T17:49:49.791639Z",
			"lastEventTime": "2020-05-12T17:49:49.791639Z",
			"lastUpdateTime": "2020-05-12T17:51:43.53Z",
			"resolvedTime": null,
			"machineId": "4d0ddab627ea0e54dc6ea75196b4b39db297f110",
			"computerDnsName": "testmachine1",
			"aadTenantId": "2978caa4-d66c-4c7e-9e30-40034eb7c6f3",
			"relatedUser": null,
			"comments": [],
			"evidence": []
		}
	]
}

Any new API calls could use a field of choice by the user. The field type could also be a choice for the user between int and date (with a date format if needed).

We then need a way to define where the value is to be inserted, in this example it would be a URL parameter in the GET request:

Based on the newest incidentId of the newest event:
https://api-eu.securitycenter.windows.com/api/alerts?$filter=incidentId+gt+2

Based on the newest alertCreationTime of the newest event:
https://api-eu.securitycenter.windows.com/api/alerts?$filter=alertCreationTime+gt+2020-05-12T17:50:48.5153821Z

Metadata

Metadata

Assignees

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions