Implement event history insert and insertIfNotExists consequence support#1121
Conversation
| /// - data: a `Traversable` object used for token resolution | ||
| private func processSchemaConsequence(consequence: RuleConsequence, processedEvent: Event, data: Traversable) { | ||
| guard let schema = consequence.schema else { | ||
| Log.error(label: LOG_TAG, "(\(self.name)) : Unable to process a Schema Consequence, 'schema' is missing from 'details'") |
There was a problem hiding this comment.
nit: As per the definition of schema consequence type, the id and data are also mandatory fields. Wondering if we should have that check here and fast fail
There was a problem hiding this comment.
Thank you! That's a great point - I believe Messaging does this as well (through the decoding mechanism, and having those properties as non-optional). I've updated the check for all three properties to be here
sbenedicadb
left a comment
There was a problem hiding this comment.
looking good so far, a few things to address
| public func recordHistoricalEvent(_ event: AEPCore.Event, handler: ((Bool) -> Void)?) { | ||
| // no-op | ||
| } | ||
|
|
||
| public func historicalEventExists(_ event: AEPCore.Event, handler: @escaping (Bool) -> Void) { | ||
| // no-op | ||
| } | ||
|
|
There was a problem hiding this comment.
i'd recommend adding two things to the new methods in this mock:
- call the handler with a value that can be provided by the caller.
- keep a boolean value that keeps track of when the method has been called
There was a problem hiding this comment.
Thank you! Added support for both points
| /// - event: the `Event` to be recorded in the Event History database | ||
| /// - handler: called with a `Bool` indicating a successful database insert | ||
| func recordHistoricalEvent(_ event: Event, handler: ((Bool) -> Void)? = nil) { | ||
| eventHistory?.recordEvent(event, handler: handler) |
There was a problem hiding this comment.
should handler be called with false here if eventHistory is nil?
There was a problem hiding this comment.
Thank you that's a great point, I think it should. I noticed that getHistoricalEvents also has potential for a nil eventHistory instance, but doesn't have a fallback completion handler call so I added ?? handler([]). What do you think?
| } | ||
|
|
||
| func recordHistoricalEvent(_ event: Event, handler: ((Bool) -> Void)?) { | ||
| guard let eventHub = eventHub else { return } |
There was a problem hiding this comment.
same question here, should we call the handler?
| /// - handler: called with a `Bool` indicating a successful database insert. | ||
| func recordEvent(_ event: Event, handler: ((Bool) -> Void)? = nil) { | ||
| guard event.eventHash != 0 else { | ||
| Log.debug(label: LOG_TAG, "Failed to record event in history - event hash is 0") |
| /// - Parameters: | ||
| /// - event: the `Event` to check for existence in the Event History database | ||
| /// - handler: called with a `Bool` indicating if an event with the same hash exists | ||
| func eventExists(_ event: Event, handler: @escaping (Bool) -> Void) { |
There was a problem hiding this comment.
do we need this function? don't we achieve the same thing by calling getEvents?
There was a problem hiding this comment.
That's a great point - I had created this because it was quite difficult to convert an Event into the EventHistoryRequest format getEvents required in the context of the processEventHistoryOperation method. My first thought was since Event itself easily converts into a hash, but none of the EventHistory APIs accept a hash directly, to add support by providing a new API in the form of this eventExists
But on second thought, given your feedback, if the issue was the difficulty of converting Event into the EventHistoryRequest format, then simply adding support for that conversion (in the new Event+EventHistoryRequest.swift) and then leveraging the existing getEvents:
- Avoids creating a narrow and brittle use case API in EventHistory
- Adds support for other use cases that would need the Event ->
EventHistoryRequestin the future
What do you think of this updated approach?
There was a problem hiding this comment.
perfect - i think that's a great plan 👍
| @@ -61,6 +61,7 @@ class EventHistoryDatabase { | |||
| dispatchQueue.async { | |||
There was a problem hiding this comment.
not your code, but can we add a capture of [weak self] for any of these async blocks where we are using self so we can prevent potential leaking?
There was a problem hiding this comment.
Yes! I added the [weak self] guarded usage, added log messages for that case, and updated the log prefix to be static
| /// - data: a `Traversable` object used for token resolution | ||
| private func processSchemaConsequence(consequence: RuleConsequence, processedEvent: Event, data: Traversable) { | ||
| guard let schema = consequence.schema, let _ = consequence.data, let _ = consequence.detailID else { | ||
| Log.error(label: LOG_TAG, "(\(self.name)) : Unable to process Schema Consequence with ID: \(consequence.id). A Schema Consequence requires 'schema', 'data', and 'id' properties.") |
There was a problem hiding this comment.
let's change this to a warn please
| /// - consequence: the RuleConsequence which contains the event history operation details | ||
| /// - processedEvent: the event that triggered the rule | ||
| /// - data: a `Traversable` object used for token resolution | ||
| private func processEventHistoryOperation(consequence: RuleConsequence, processedEvent: Event, data: Traversable) { |
There was a problem hiding this comment.
let's get the magic strings out of this method:
operationkeys~content
There was a problem hiding this comment.
Thank you! Extracted them to the top of the file, unless there's a better place to put them you had in mind?
| /// - data: a `Traversable` object used for token resolution | ||
| private func processEventHistoryOperation(consequence: RuleConsequence, processedEvent: Event, data: Traversable) { | ||
| guard let schemaData = consequence.data, let operation = schemaData["operation"] as? String else { | ||
| Log.error(label: LOG_TAG, "(\(self.name)) : Unable to process an EventHistoryOperation Consequence with ID: \(consequence.id). Required 'operation' field missing from 'detail.data'.") |
| if success { | ||
| self.extensionRuntime.dispatch(event: eventToRecord) | ||
| } else { | ||
| Log.error(label: self.LOG_TAG, "(\(self.name)) : Failed to record event in history with operation '\(operation)'") |
…xists API and update usages
…Event method in TestableExtensionRuntime
timkimadobe
left a comment
There was a problem hiding this comment.
Thanks so much for the review @sbenedicadb! Updated based on feedback
| /// - Parameters: | ||
| /// - event: the `Event` to check for existence in the Event History database | ||
| /// - handler: called with a `Bool` indicating if an event with the same hash exists | ||
| func eventExists(_ event: Event, handler: @escaping (Bool) -> Void) { |
There was a problem hiding this comment.
That's a great point - I had created this because it was quite difficult to convert an Event into the EventHistoryRequest format getEvents required in the context of the processEventHistoryOperation method. My first thought was since Event itself easily converts into a hash, but none of the EventHistory APIs accept a hash directly, to add support by providing a new API in the form of this eventExists
But on second thought, given your feedback, if the issue was the difficulty of converting Event into the EventHistoryRequest format, then simply adding support for that conversion (in the new Event+EventHistoryRequest.swift) and then leveraging the existing getEvents:
- Avoids creating a narrow and brittle use case API in EventHistory
- Adds support for other use cases that would need the Event ->
EventHistoryRequestin the future
What do you think of this updated approach?
| /// - event: the `Event` to be recorded in the Event History database | ||
| /// - handler: called with a `Bool` indicating a successful database insert | ||
| func recordHistoricalEvent(_ event: Event, handler: ((Bool) -> Void)? = nil) { | ||
| eventHistory?.recordEvent(event, handler: handler) |
There was a problem hiding this comment.
Thank you that's a great point, I think it should. I noticed that getHistoricalEvents also has potential for a nil eventHistory instance, but doesn't have a fallback completion handler call so I added ?? handler([]). What do you think?
| } | ||
|
|
||
| func recordHistoricalEvent(_ event: Event, handler: ((Bool) -> Void)?) { | ||
| guard let eventHub = eventHub else { return } |
| @@ -61,6 +61,7 @@ class EventHistoryDatabase { | |||
| dispatchQueue.async { | |||
There was a problem hiding this comment.
Yes! I added the [weak self] guarded usage, added log messages for that case, and updated the log prefix to be static
| /// - data: a `Traversable` object used for token resolution | ||
| private func processSchemaConsequence(consequence: RuleConsequence, processedEvent: Event, data: Traversable) { | ||
| guard let schema = consequence.schema, let _ = consequence.data, let _ = consequence.detailID else { | ||
| Log.error(label: LOG_TAG, "(\(self.name)) : Unable to process Schema Consequence with ID: \(consequence.id). A Schema Consequence requires 'schema', 'data', and 'id' properties.") |
| /// - data: a `Traversable` object used for token resolution | ||
| private func processEventHistoryOperation(consequence: RuleConsequence, processedEvent: Event, data: Traversable) { | ||
| guard let schemaData = consequence.data, let operation = schemaData["operation"] as? String else { | ||
| Log.error(label: LOG_TAG, "(\(self.name)) : Unable to process an EventHistoryOperation Consequence with ID: \(consequence.id). Required 'operation' field missing from 'detail.data'.") |
| if success { | ||
| self.extensionRuntime.dispatch(event: eventToRecord) | ||
| } else { | ||
| Log.error(label: self.LOG_TAG, "(\(self.name)) : Failed to record event in history with operation '\(operation)'") |
| /// - consequence: the RuleConsequence which contains the event history operation details | ||
| /// - processedEvent: the event that triggered the rule | ||
| /// - data: a `Traversable` object used for token resolution | ||
| private func processEventHistoryOperation(consequence: RuleConsequence, processedEvent: Event, data: Traversable) { |
There was a problem hiding this comment.
Thank you! Extracted them to the top of the file, unless there's a better place to put them you had in mind?
| public func recordHistoricalEvent(_ event: AEPCore.Event, handler: ((Bool) -> Void)?) { | ||
| // no-op | ||
| } | ||
|
|
||
| public func historicalEventExists(_ event: AEPCore.Event, handler: @escaping (Bool) -> Void) { | ||
| // no-op | ||
| } | ||
|
|
There was a problem hiding this comment.
Thank you! Added support for both points
| var tokenDictionary: [String: Any?] = [:] | ||
| if !tokenKeys.isEmpty { | ||
| for key in tokenKeys { | ||
| tokenDictionary[key] = "\(LaunchRulesEngine.LAUNCH_RULE_TOKEN_LEFT_DELIMITER)\(key)\(LaunchRulesEngine.LAUNCH_RULE_TOKEN_RIGHT_DELIMITER)" |
There was a problem hiding this comment.
On L173, we're traversing through the entire rule consequence and already parsing all tokens inside {%...%} and replacing them with values from event data.
Wondering if we should have the place where the rule is authored add in whatever is inside the keys part with those delimiters so we don't do this manual addition here and this looks consistent with the other attach and modify consequence . The rule we receive from the server would look like this
{
"id": "123",
"type": "schema",
"detail": {
"id": "abc",
"schema": "https://ns.adobe.com/personalization/eventHistoryOperation",
"data": {
"operation": "insert",
"keys": [
"keyToWriteToEventHistory: {%keyInEventData%}"
],
"content": {
"key": "value"
}
}
}
}
Thoughts?
There was a problem hiding this comment.
I think you're right - in the proposal I had seen an example keys with token values not already wrapped in the token delimiters, so I implemented based on that. However, given how attach and modify consequences work today and expect the values to be replaced as tokens to already be wrapped in the delimiters, I agree that we should require the rule author to do the wrapping
But with the current logic and given that in the proposal, keys is an array of Strings and can have a mix of event keys and token keys, if the replacement happens on L173, how do we know which are event keys and which are token keys with their values resolved?
Ex:
["event.key", "{%~type%}"] --(resolved on L173)--> ["event.key", "com.adobe.eventType.analytics"]
Then in the logic of the schema consequence to pull from the event data using the keys array, how do we:
- Know which were originally event keys and which were tokens and
- Know what the original token name was, since tokens need to be set in the hash with the token name as key and resolved value as value?
There was a problem hiding this comment.
I believe the LaunchTokenFinder already has the logic for this today. If the value starts with~followed by a known type, like type, source, state, the appropriate value is read. If~is followed by an unknown string, we attempt to read from the event data.
I might have misunderstood your question. I get your point now. We need to pass the event data keys in the mask array to write to event history. That's what you were getting at right? We might need move token keys to content part and keep event keys in the keys :
{
"id": "123",
"type": "schema",
"detail": {
"id": "abc",
"schema": "https://ns.adobe.com/personalization/eventHistoryOperation",
"data": {
"operation": "insert",
"keys": ["keyInEventData"],
"content": {
"key": "value",
"eventType": "{%~type%}",
"keyToWriteToEventHistory": "{%~tokenToBeReplacedFromEventData%}
}
}
}
}
There was a problem hiding this comment.
Adding the tokens to content would also avoid us calling EventDataMerger.merge twice
There was a problem hiding this comment.
Yes, I think moving the token keys to content would solve this. @sbenedicadb what do you think?
There was a problem hiding this comment.
After discussion with Steve, I think we landed on removing support for the keys property in the consequence for now, at least until a clear use case for the property comes up - I've updated the implementation to reflect this, please let me know what you think!
spoorthipujariadobe
left a comment
There was a problem hiding this comment.
LGTM! Lets add tests to cover the new rule consequences
* Implement event history insert and insertIfNotExists consequence support (#1121) * Implement event history insert and insertIfNotExists consequence support * Add method stubs * Update API dump * Clean up TestableExtensionRuntime * Add early exit check for schema consequence, update log messages and levels * Dispatch consequence event for insert and insertIfNotExists success * Add Event to EventHistoryRequest conversion support and remove eventExists API and update usages * Add nil case handler calls for recordHistoricalEvent and getHistoricalEvents * Add weak self to closures and update LOG_PREFIX to static instance * Update log levels from error to warning * Extract strings from dictionary access logic into constants * Add method call and result configuration support for recordHistoricalEvent method in TestableExtensionRuntime * Fix early exit case for no keys specified to record in consequence * Remove support for the `keys` property * Add back content empty check early exit, improve logging detail * Extract repeated log info into variable * Update eventHistoryOperation logic to align with Android, clean up unused constants (#1133) * Create protocols for EventHistory and EventHistoryDatabase and update EventHub to use dependency injection for EventHistory Update MockEventHistoryDatabase to use new EventHistoryDatabaseService protocol Update EventHistory and EventHistoryDatabase to conform to their respective protocols Update MockEventHistory to conform to EventHistoryService protocol and fix implementation Fix EventHubEventHistoryTests to use the instantiated eventhub instance not the shared one # Conflicts: # AEPCore.xcodeproj/project.pbxproj # AEPCore/Mocks/PublicTestUtils/MockEventHistoryDatabase.swift # AEPCore/Tests/EventHubTests/EventHubEventHistoryTests.swift # AEPCore/Tests/TestHelpers/MockEventHistory.swift * Fix merge * Fix event history operation `insertIfNotExists` logical error and race with insert operation (#1134) * Fix insertIfNotExists escaping closure operation race with insert operation * Remove max time deadline for getHistoricalEvents in historical event consequence processing Add check for database error value in CONSEQUENCE_EVENT_HISTORY_OPERATION_INSERT_IF_NOT_EXISTS logic and improve logging around this value * Remove unused constant * Split log lines * Update EventHub's eventHistory reference from var to let * Update protocol names and add storage property to EventHistoryProvider Update EventHistory init to accommodate this read only access to the backing storage * Delete incorrectly renamed file * Pod update * Add test coverage and support implementation for EventHub EventHistory related method tests * Add test coverage for new Event toEventHistoryRequest * Add documentation notes to keep initializers consistent * Create protocols for EventHistory and EventHistoryDatabase and update EventHub to use dependency injection for EventHistory Update MockEventHistoryDatabase to use new EventHistoryDatabaseService protocol Update EventHistory and EventHistoryDatabase to conform to their respective protocols Update MockEventHistory to conform to EventHistoryService protocol and fix implementation Fix EventHubEventHistoryTests to use the instantiated eventhub instance not the shared one * Fix merge * Fix merge * Fix merge - remove old EventHistoryDatabaseService.swift and EventHistoryService.swift * Update MockEventHistory to conform to EventHistoryProvider * Fix rebase errors * Update Event_EventHistoryRequestTests to better test nested structures * Update EventHubEventHistoryTests to clean up event names and descriptions * Run pod update * Create rules engine test base class and implement test cases for historical event operation # Conflicts: # AEPCore.xcodeproj/project.pbxproj * Clean up code * Add event record, dispatch, and payload validation * Clean up comments * Update insertIfNotExists test to check the database error case Remove unneeded getHistoricalEvents delay helper in mock * Fix project file inclusion post rebase * Run pod update * Implement the mostRecent search type handler * Fix logical error in description * Create tests for most recent condition cases * Add functional tests for mostRecent condition working in rules engine * Add test cases to EventHistoryTests to improve coverage for the "more than one" event case Update MockEventHistoryDatabase to enable this testing * Update test case to differentiate return values * Update search type logic to handle setting value more elegantly and remove default value logging Replace search type operand handling with switch Refactor getHistoricalEventCount with switch instead of nested if else and add early exit guard for unsupported type Fix doc error * Create unit tests for JSONCondition's getHistoricalEventCount method * Add back added test file * Fix missing dispatch schema consequence event for non-historical event operation * Fix project structure * Add support for array flattening and add and update tests * Add ability for all_url to not use the new array flattening logic for backwards compatibility * Update examples for clarity * Apply swift formatting * Add dictionary shape test and update test case comment * Add test cases for flattenArrays false, tighten test case expectations for some cases * Fix NSNull values filtering in the FNV1A32 hash generation and test parity with Android * Update historical condition consequence to use the generateConsequenceEvent method to dispatch consequence Update test cases * Update test cases to validate new recorded and dispatched events * Update event history getEvents logs to trace * Add resource class to install_dependencies * Update to Xcode 15.3 * Update iOS/tvOS devices to 17.4 * Fix missing Xcode version upgrade * Fix files lost in merge --------- Co-authored-by: Steve Benedick <sbenedic@adobe.com>
Description
This PR implements support for the new
insertandinsertIfNotExistsconsequence support in the Core RulesEngine, leveraging the existing rules condition evaluation logic to trigger these consequences.The consequence handling has been updated to handle the new
schemaconsequence type, andhttps://ns.adobe.com/personalization/eventHistoryOperationschema value.At a high level, the consequence object has three high level properties:
Determines the type of database operation to be performed on Event History by the SDK.
- Insert: Inserts a record in the Event History Database.
- Insert if not exists: Inserts a record in the Event History Database only if there is not already a matching record with a matching hash value.
Provided key-value pairs will be added to the string used to generate the event history hash.
If the value represents a token that the SDK knows how to resolve, the resolved value will be used (such as shared state values, ~timestampu, etc.).
An array of strings that designate which keys in the event will be used for generating the event history hash.
The primary logic for this new support is handled in the new
LaunchRulesEngineprocessEventHistoryOperationmethod:operationexists (which could be eitherinsertorinsertIfNotExists, but does not strictly require these two values for future expansion)keysare separated into two groups - event data keys and token keys: -~(ex:~sdkver)["~sdkver": "{%~sdkver%}"]2 This dictionary is resolved using the
TokenFinderused to resolve the original rules conditioncontentkey value pairs are merged in (thecontentkey value pairs' tokens were already replaced by the overall consequence resolution)keys(that is, event data keys and token keys) andcontentinsertIfNotExists, first check that the hash for the event does not already existeventExistshas been created in inEventHistoryto support easily checking for the hash value instead of having to extract all the key value pairs from the flattened map as is required by the existinggetEventsAPI'sEventHistoryRequestinput format requirementQuestions for reviewers
processEventHistoryOperationuses direct string keys - how do we feel about this? Should we use constants here instead? If so where should they live?:evaluateRulesloop, because the EventHistory methods it depends on are async methods. Is the approach taken acceptable? Should we also be keeping track ofdispatchChainCounthere?Related Issue
Motivation and Context
How Has This Been Tested?
Screenshots (if appropriate):
Types of changes
Checklist: