Skip to content

Implementation of Real Time Config#9857

Merged
alanorozco merged 59 commits intoampproject:masterfrom
google:frizz-real-time-config
Jul 19, 2017
Merged

Implementation of Real Time Config#9857
alanorozco merged 59 commits intoampproject:masterfrom
google:frizz-real-time-config

Conversation

@bradfrizzell
Copy link
Copy Markdown
Contributor

@bradfrizzell bradfrizzell commented Jun 12, 2017

Beginning implementation of RTC.

<title>Ad examples</title>
<link rel="canonical" href="http://nonblocking.io/" >
<meta name="viewport" content="width=device-width,minimum-scale=1,initial-scale=1">
<style amp-custom>
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you really need all of this style and extensions?

return '';
}

const rtcConfig = tryParseJson(
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be within getPageLevelParameters_. Also, correct me if I'm wrong but this would cause the RTC request to be sent for every slot instead of once per page. We should refactor the code so that getPageLevelParameters_ returns a cached value if already calculated which will include ensuring RTC is only sent once.

['108809080']));
.then(p => {
pageLevelParameters = p;
return googlePageParameters(this.win, this.getAmpDoc(), startTime);
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm confused, getPageLevelParameters_ calls googlePageParameters so why do you need to do it?

/** @private {?Promise} */
let sraRequests = null;

/** @private {?Promise} */
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Type here can be more precise (e.g. ?Promise<!Object<string,string>>)

return '';
}

const ampRtcPageElement = document.getElementById('amp-rtc');
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do not worry about other networks, assume only doubleclick

rtcConfig = tryParseJson(ampRtcPageElement.innerHTML);
if (rtcConfig && typeof rtcConfig == 'object'
&& !!rtcConfig['doubleclick']) {
rtcRequestPromise = this.sendRtcRequestPromise_(/** @type {!Object} */(
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this can just be sendRtcRequest_ (no need to include promise)

googleAdUrl(this, DOUBLECLICK_BASE_URL, startTime,
Object.assign(this.getBlockParameters_(), pageLevelParameters),
['108809080']));
const pageLevelParametersPromise = getPageLevelParameters_(
Copy link
Copy Markdown
Contributor

@keithwrightbos keithwrightbos Jun 28, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RTC modifies block level parameters so you don't actually have to wait on page level before merging. You can move the merge directly into sendRTCRequest. This has the added benefit of allowing you to add other parameters (e.g. time to get RTC response) without putting that logic in getAdUrl.

Also I don't think how you have this will work as this.jsonTargeting_ is unique per block but you only have one promise that merges. So the flow actually needs to be that the RTC promise just returns back the RTC response and then you call merge on every block. Does that make sense?

// Default is to send request without RTC, so only reject if
// pub explicitly says not to.
if (rtcConfig['doubleclick']['sendAdRequestOnFailure'] === false) {
return Promise.reject();
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are 3 failure conditions of which any failure should prevent ad request if indicated by pub:

  • network failure
  • timeout
  • malformed response

This is addressing the last item but I would argue that only malformation we care about is failure to JSON parse which we can handle as part of sendRTCRequest flow

['108809080']));
const pageLevelParametersPromise = getPageLevelParameters_(
this.win, this.getAmpDoc(), startTime);
const blockParameters = this.getBlockParameters_();
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You don't actually need blockParameters for RTC. You only need populateAdUrlState to have set this.jsonTargeting. I suggest we structure as follows:

  • getPageParams
  • populateAdUrlState
  • sendRtcRequest which modifies this.jsonTargeting_ and return an object mapping to be merged into the end url. sendRtcRequest contains ALL RTC logic (perhaps rename to executeRTC?) from merge to timeout to reject if pub indicates no ad request on failure
  • getBlockParams (AFTER RTC completion)
  • Object.assign block, RTC, page (note that order indicates order of url parameters!)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

}
const errorUrl = rtcDblckConfig['errorReportingUrl'];
const endpoint = rtcDblckConfig['endpoint'];
const noCache = (rtcDblckConfig['noCache'] === true);
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider looking for other fields in the object to point out typos to pubs.

const DOUBLECLICK_BASE_URL =
'https://securepubads.g.doubleclick.net/gampad/ads';

/** @const {Object} */
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can make this @enum {string}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

BAD_ENDPOINT: 'Bad RTC Endpoint',
};
/** @const {number} */
const RTC_TIMEOUT = 1000;
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add MS to indicate its milliseconds

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

dev().warn(TAG, `Frame already exists, sra: ${this.useSra}`);
return '';
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: remove extra, empty lines

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

// TODO(keithwrightbos): SRA blocks currently unnecessarily generate full
// ad url. This could be optimized however non-SRA ad url is required to
// fallback to non-SRA if single block.
this.populateAdUrlState();
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You need to call populateAdUrlState before sendRtcRequest to ensure that this.jsonTargeting_ has been populated

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

}

const ampRtcPageElement = document.getElementById('amp-rtc');
let rtcConfig;
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if (!ampRtcPageElement) return Promise.resolve();
let rtcConfig;
let endPoint;
// NOTE usage of textContent instead of innerHTML
if (!isObject(rtcConfig = tryParseJson(ampRtcPageElement.textContent)) || !(endPoint = rtcConfig['endPoint']) || typeof endPoint != 'string' || !isSecureUrl(endPoint)) {
user().warn(TAG, 'invalid RTC config...');
return Promise.resolve();
}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

/**
* Sends RTC request as specified by rtcConfig. Returns promise which
* resolves to the targeting informtation from the request response.
* If the publisher has specified that noCache == true, then we will only
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Whether or not the response was cached should be controlled by the end point via cache headers. I think it makes more sense to have an option that disables state while revalidate behavior (e.g. staleRevalidateDisabled: true). The behavior is that the first request has no special cache headers and if SWR is not disabled, we send a second request with no-cache option

const endpoint = rtcConfig['endpoint'];
const noCache = (rtcConfig['noCache'] === true);

if (!endpoint) {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note my previous comment does not cancel the ad request if invalid config. I think this makes more sense as the pub clearly misconfigured and we should default to ensuring they get ad revenue

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ack


return rtcPromise = timerFor(window).timeoutPromise(
RTC_TIMEOUT, rtcRequest).then(res => {
// TODO :: Need to add the time on to the ad request.
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suggest we rename this function to executeRTC and have it return Promise<!Object<string,string>> allowing for custom parameters to be merged into the ad url (e.g. time delay). Note that this will be page level by definition.


if (!noCache) {
// Repopulate the cache.
xhrFor(this.win).fetchJson(endpoint, {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that given the current flow, we will possibly send the SWR request before the first request has returned or possibly errored due to non-200 response. Let's change this flow so that sending the cache request is actually dependent on the first rtc request promise. Something like:

const rtcRequest = this.executeRTC_();
if (!swrDisabled) {
rtcRequest.then(() => {
// send rtc request with no-cache option
});
}
return rtcPromise = timerFor(window).timeoutPromise(
RTC_TIMEOUT, rtcRequest).then(res => {
// merge into this.jsonTargeting_ and generate ad url here
return customParams;
});

// TODO(tdrl): Check amp-ad registry to see if they have this already.
if (!a4aRegistry[type] ||
this.win.document.querySelector('meta[name=amp-3p-iframe-src]') ||
(this.win.document.querySelector('meta[name=amp-3p-iframe-src]') &&
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not a fan of the fact that RTC (which is doubleclick specific) is cluttering up amp-ad. We should rethink how to do this later...

});
});

it('should not send RTC if url invalid', () => {});
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

note: still filling these cases in as I go. just pushed out to get the other changes out there.

// Non-200 status codes are forbidden for RTC.
// TODO: Add to fetchResponse the ability to
// check for redirects as well.
if (res.status != 200) {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about 204 no content? Other 200 codes?

return null;
}
return res.text().then(text => {
if (text == '') {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add a comment indicating that empty is assumed to indicate ok response with nothing to merge. Can also just do if (!text)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

if (!r) {
return this.shouldSendRequestWithoutRtc(
'Bad response');
} else if (typeof r == 'number') {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I prefer a consistent object type. How about instead you just check if !r.rtcResponse here?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

const rtcResponse = r.rtcResponse;
if (!!rtcResponse['targeting']) {
this.jsonTargeting_['targeting'] = deepMerge(
this.jsonTargeting_['targeting'] || {},
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No need to call deepMerge if this.jsonTargeting_['targeting'] exists, right? Can just assign it. Not sure how much that saves in terms of performance

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assume you meant that we don't need to call deepMerge if this.jsonTargeting['targeting'] does not exist. I'll change it

const rtcSuccess = values[1]['success'];
if (rtcTotalTime) {
parameters['artc'] = rtcTotalTime;
parameters['ard'] = parseUrl(rtcConfig['endpoint']).hostname;
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm thinking a cleaner interface is to just have executeRtc return parameters to be added.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done, switched to using Object.assign too. even cleaner

}

const rtcResponse = r.rtcResponse;
if (!!rtcResponse['targeting']) {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider creating helper function that takes keyName (e.g. targeting) to reduce duplicate code

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

shouldSendRequestWithoutRtc(errMessage) {
user().error(TAG, errMessage);
let rtcTotalTime;
// Have to use match instead of == because AMP
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this comment still apply?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes.

* model.
* The targeting info from the RTC updates the targeting info on
* this object within mergeRtc.
* @return {?Promise<?number|?string>} The time the RTC callout took
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to update return type here and mergeRtc

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

const pageLevelParameters = values[0];
let parameters = Object.assign(
this.getBlockParameters_(), pageLevelParameters);
if (values[1] && values[1]['artc']) {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Anyway you could not require looking for 'artc'?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yea don't actually need it. Removed.

const pageLevelParameters = values[0];
let parameters = Object.assign(
this.getBlockParameters_(), pageLevelParameters);
if (!!values[1]) {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you don't actually need this check as assigning null/undefined has no effect. You can change this to:

const parameters = Object.assign(this.getBlockParameters_(), values[1], pageLevelParameters);

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

if (!isObject(rtcConfig) || !(endpoint = rtcConfig['endpoint']) ||
typeof endpoint != 'string' || !isSecureUrl(endpoint)) {
user().warn(TAG,
`invalid RTC config: ${ampRtcPageElement.textContent}`);
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's add that we're sending the ad request anyway: 'invalid RTC config, sending ad request w/o callout: ${...}'

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

}
let rtcTotalTime;
const disableSWR = (
rtcConfig['disableStaleWhileRevalidate'] === true);
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you consider edge case values such as 'true', 1, and '1'? Ideally there would be a helper function to ensure we are consistent across the codebase (perhaps one already exists?). See sendAdRequestOnFailure you have below.

Also let's move this directly into the if condition below, no need to create variable.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll add 'true' as valid here too, but I'm not sure adding 1 and '1' is necessarily correct.

Services.xhrFor(this.win).fetchJson(
endpoint, {credentials: 'include'}).then(res => {
rtcTotalTime = Date.now() - startTime;
if (!disableSWR) {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add a comment that SWR should be disabled if the RTC is not cachable (no cache header). Can we read the cache header from the response? Also something about Cache-Control not being supported in IE?

!!this.jsonTargeting_[key] ?
deepMerge(this.jsonTargeting_[key],
rtcResponse[key]) :
rtcResponse[key];
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This indentation looks off...

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

rtcResponse[key];
}
};
mergeHelper('targeting');
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we make a whitelist of mergable properties? Then you can just iterate and makes adding others easier in the future

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not totally sure what a whitelist is supposed to look like. I changed it to use a list and that iterate over it. Is that good for now at least?

// does not include the time to merge.
return Promise.resolve({
artc: r.rtcTotalTime,
ati: 2,
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's make an enum for Ati values to make the code clearer

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

/** @private {?JsonObject|undefined} */
let rtcConfig = null;

/** @private {Object} */
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Private @enum {number}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

rtcConfig['disableStaleWhileRevalidate'] === true);
if (rtcConfig['disableStaleWhileRevalidate'] != undefined &&
typeof rtcConfig['disableStaleWhileRevalidate'] != 'boolean') {
user().warn('RTC disableStaleWhileRevalidate must be a boolean.');
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Include the value they set in the message. Also indentation of typeof looks off

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

const init = {
credentials: 'include',
};
// If stale-while-revalidate is disabled, our first RTC callout
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wait why? I think my last comment might have been confusing. The flow should be:

  • always send RTC with ability to read from cache
  • if response indicates not to use SWR OR response indicates it was not cached, then do NOT make SWR. The latter might be difficult depending on if you can read cache headers clientside

} else if (!r.rtcResponse && r.success) {
return Promise.resolve({
artc: r.rtcTotalTime,
ati: 2,
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use your new enum here, can you also add a comment indicating that this is the case where you got an empty response so no need to merge

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

if (errMessage.match(/^timeout/)) {
rtcTotalTime = -1;
}
return rtcConfig['sendAdRequestOnFailure'] !== false ?
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add typeof warn?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

rtcResponse[key];
}
};
['targeting', 'categoryExclusions'].forEach(key => {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: can be forEach(key => mergeHelper) or move content of mergeHelper into forEach

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

/**
* @param {string} member
* @param {string} expectedType
*/
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: Appears that you're only verifying boolean types so no need to actually pass it. Suggest rename to: logIfRtcConfigMemberNotBoolean(member)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wanted to make it extensible so I can use for other fields in the config

this.win.document.querySelector('meta[name=amp-3p-iframe-src]')) ||
// Note that predicate execution may have side effects.
this.win.document.querySelector('meta[name=amp-3p-iframe-src]') &&
!this.win.document.getElementById('amp-rtc')) ||
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please bring back to the comment: // Note that predicate...


const pageLevelParametersPromise = getPageLevelParameters_(
this.win, this.getAmpDoc(), startTime);
const rtcRequestPromise = this.executeRtc_();
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually one more thing: can you add an experiment so that we could turn this off easily if needed?

const rtcRequestPromise = isExperimentOn('disable-rtc') ? Promise.resolve({}) : this.executeRtc_();

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants