Skip to content

improve(price-feed): Add more error details to aggregator PriceFeeds about which sub-pricefeed is missing data#2426

Merged
nicholaspai merged 29 commits intomasterfrom
npai/patch-historical-pricefeed
Feb 2, 2021
Merged

improve(price-feed): Add more error details to aggregator PriceFeeds about which sub-pricefeed is missing data#2426
nicholaspai merged 29 commits intomasterfrom
npai/patch-historical-pricefeed

Conversation

@nicholaspai
Copy link
Copy Markdown
Member

@nicholaspai nicholaspai commented Jan 19, 2021

Summary

  • Part of Bot upgrades related to historical price querying and WDF #2419 addressing historical price feed debug logs. The main issue this PR addresses is that the the aggregator pricefeeds like BasketSpread and Medianizer will return invalid for a price if ANY of their constituent price feeds throw an error, but usually only one price feed is failing. This PR adds logic to the aggregator pricefeeds to query their constituent pricefeeds for additional data.
  • Fix TransferSynthetic and TransferCollateral scripts to work with ABI versioning logic

Details

Example new logs for the BasketSpreadPriceFeed:
Screen Shot 2021-01-21 at 12 36 17

Signed-off-by: Nick Pai <npai.nyc@gmail.com>
Signed-off-by: Nick Pai <npai.nyc@gmail.com>
Signed-off-by: Nick Pai <npai.nyc@gmail.com>

// Filter out events where price is null.
this.events = events.filter(e => e.price !== null);
this.events = this.events.filter(e => e.price !== null);
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Random bug I found, don't think this has really affected anything yet

Signed-off-by: Nick Pai <npai.nyc@gmail.com>
@coveralls
Copy link
Copy Markdown

coveralls commented Jan 19, 2021

Coverage Status

Coverage remained the same at 93.284% when pulling 6db18ee on npai/patch-historical-pricefeed into e6a54e6 on master.

Signed-off-by: Nick Pai <npai.nyc@gmail.com>
Signed-off-by: Nick Pai <npai.nyc@gmail.com>
Signed-off-by: Nick Pai <npai.nyc@gmail.com>
Signed-off-by: Nick Pai <npai.nyc@gmail.com>
Signed-off-by: Nick Pai <npai.nyc@gmail.com>
@nicholaspai nicholaspai requested review from chrismaree, daywiss and mrice32 and removed request for chrismaree January 20, 2021 19:55
@nicholaspai nicholaspai marked this pull request as ready for review January 20, 2021 19:55
Copy link
Copy Markdown
Member

@mrice32 mrice32 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great! A few comments and questions.

Comment on lines +121 to +124
if (priceFeed instanceof MedianizerPriceFeed) {
// MedianizerPriceFeed.debugHitoricalData returns an array of error details so we will
// concat its error log array.
priceFeedErrorDetails = priceFeedErrorDetails.concat(priceFeed.debugHistoricalData(time));
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not just have every price feed return a string as its own error message and higher-level price feeds just join the strings with separators so you don't have to detect the medianizer price feed specifically?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a good idea, originally I wanted to shape the errors as an array of strings, but I don't think the array of strings vs concatenated string makes much difference to the end user.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

However, I still like the idea of the higher level price feeds defining the error strings based on the lower level price feed's UUID's. Ultimately, this approach removes the need for each lower level price feed to implement debugHistoricalData, but the tradeoff is that the higher level pricefeeds should check if a lower level price feed has implemented debugHistoricalData first before it constructs an error string for the lower price feed. Look at this commit for example.

this.priceFeeds.map(priceFeed => {
const hasPrice = priceFeed.getHistoricalPrice(time);
if (!hasPrice) {
priceFeedErrorDetails.push(`PriceFeed down with UUID: ${priceFeed.uuid}`);
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might be useful to include the time that's unavailable in these since it may be particular times that are broken as opposed to the entire feed being down?

return this._getSpreadFromBasketPrices(experimentalPrices, baselinePrices, denominatorPrice);
}

debugHistoricalData(time) {
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a random idea -- have you thought about returning an error message from the getHistoricalData method itself? That way, we wouldn't need to add a new method to the interface. The logic you have here would just be baked into the normal method. We could even say the second element is optional, so if you don't provide it, you're just failing to provide the caller with info about the error. Something like:

return [price, errorString];

errorString could be null in the success case and price could be null in the error case?

An even more nuclear and invasive option is to begin allowing these price feeds to throw errors and relying on the calling code to catch those errors?

What do you think?

Copy link
Copy Markdown
Member Author

@nicholaspai nicholaspai Jan 21, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this would create more churn by changing the return interface of getHistoricalPrice(), so I came up with a solution that doesn't affect the price feed interface. Do you think this is worth implementing? I'm not 100% happy with my solution but I think its non-invasive and in the worst case doesn't change anything for the end user.


debugHistoricalData(time) {
let priceFeedErrorDetails = [];
this.allPriceFeeds.map(priceFeed => {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is just a nit, but you are using map but also pushing data into the priceFeedErrorDetails array. Consider changing this to forEach for side effects like this, or using reduce (removing need for priceFeedErrorDetails) since you are potentially changing length of array.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea, I am not using map how you are supposed to :)


debugHistoricalData(time) {
let priceFeedErrorDetails = [];
this.priceFeeds.map(priceFeed => {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same as above, this is probably better as forEach or a reduce.

Signed-off-by: Nick Pai <npai.nyc@gmail.com>
Signed-off-by: Nick Pai <npai.nyc@gmail.com>
@nicholaspai
Copy link
Copy Markdown
Member Author

@mrice32 I responded to your comments in this commit

@daywiss I responded to your comments in this commit

Copy link
Copy Markdown
Member

@mrice32 mrice32 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've looked through the recent updates, and I was wondering if you could explain what the requirements are for someone who's writing a price feed class? Is it different if that price feed aggregates vs just reports a price from an API? It seems that we're starting to create a distinction between those two types of interfaces and place different requirements on them?

My general concern is that we're further muddying the water about what an implementer of the price feed class is expected to do. It seems that some are expected to report debug information and others are not. The reason I'm pushing on this is I think we should aim to make this interface as light and simple as possible for those trying to add price feeds since ultimately we'd like external devs to be able to do it.

@nicholaspai
Copy link
Copy Markdown
Member Author

I've looked through the recent updates, and I was wondering if you could explain what the requirements are for someone who's writing a price feed class? Is it different if that price feed aggregates vs just reports a price from an API? It seems that we're starting to create a distinction between those two types of interfaces and place different requirements on them?

If the price feed aggregates, then they should implement a debugHistoricalData method so that a client of the pricefeed can determine which of its aggregated price feeds errored out. If the price feed does not aggregate, then the pricefeed should implement a uuid so that an aggregator price feed can distinguish it. So yes there are now two classes of price feeds:

  • AggregatorPriceFeeds: apply an aggregation function across more than one BasePriceFeeds
  • BasePriceFeeds: does not rely on any other PriceFeed.

My general concern is that we're further muddying the water about what an implementer of the price feed class is expected to do. It seems that some are expected to report debug information and others are not. The reason I'm pushing on this is I think we should aim to make this interface as light and simple as possible for those trying to add price feeds since ultimately we'd like external devs to be able to do it.

I've considered extending PriceFeedInterface to create AggregatorPriceFeedInterface and BasePriceFeedInterface and every PriceFeed should implement one of the latter two. WDYT? Or is there a better way to address the core problem of this PR?

Copy link
Copy Markdown
Member

@chrismaree chrismaree left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've looked through the recent updates, and I was wondering if you could explain what the requirements are for someone who's writing a price feed class? Is it different if that price feed aggregates vs just reports a price from an API? It seems that we're starting to create a distinction between those two types of interfaces and place different requirements on them?

If the price feed aggregates, then they should implement a debugHistoricalData method so that a client of the pricefeed can determine which of its aggregated price feeds errored out. If the price feed does not aggregate, then the pricefeed should implement a uuid so that an aggregator price feed can distinguish it. So yes there are now two classes of price feeds:

  • AggregatorPriceFeeds: apply an aggregation function across more than one BasePriceFeeds
  • BasePriceFeeds: does not rely on any other PriceFeed.

My general concern is that we're further muddying the water about what an implementer of the price feed class is expected to do. It seems that some are expected to report debug information and others are not. The reason I'm pushing on this is I think we should aim to make this interface as light and simple as possible for those trying to add price feeds since ultimately we'd like external devs to be able to do it.

I've considered extending PriceFeedInterface to create AggregatorPriceFeedInterface and BasePriceFeedInterface and every PriceFeed should implement one of the latter two. WDYT? Or is there a better way to address the core problem of this PR?

I think this is a good structure, primarily because it will keep the BasePriceFeeds as simple as possible. Most products will fall into the BasePriceFeeds category and hopefully, we can re-use the aggregator logic in different feed contexts. Once this logic is finalized we should document it very clearly to explain to price feed writers what the conditions/requirements are for each kind.

The other option would be to follow more of what @mrice32 is proposing wherein each feed itself can provide debugging information, either via returning multiple values (as proposed here, via a separate log debugger/helper utility or through a logger.

We should consider taking a step back and looking at the implications of both techniques and how they generalize given we are wanting to simplify the overall process of adding price feeds.

// if the pricefeed aggregates pricefeeds, like a Medianizer or a BasketSpreadPriceFeed.
let priceFeedErrorDetails;
try {
priceFeedErrorDetails = this.priceFeed.debugHistoricalData(liquidationTime);
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is pretty great. I look forward to node >14 then we could just do .? syntax for this.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

... yeah that'd be perfect actually

@nicholaspai
Copy link
Copy Markdown
Member Author

I've looked through the recent updates, and I was wondering if you could explain what the requirements are for someone who's writing a price feed class? Is it different if that price feed aggregates vs just reports a price from an API? It seems that we're starting to create a distinction between those two types of interfaces and place different requirements on them?

If the price feed aggregates, then they should implement a debugHistoricalData method so that a client of the pricefeed can determine which of its aggregated price feeds errored out. If the price feed does not aggregate, then the pricefeed should implement a uuid so that an aggregator price feed can distinguish it. So yes there are now two classes of price feeds:

  • AggregatorPriceFeeds: apply an aggregation function across more than one BasePriceFeeds
  • BasePriceFeeds: does not rely on any other PriceFeed.

My general concern is that we're further muddying the water about what an implementer of the price feed class is expected to do. It seems that some are expected to report debug information and others are not. The reason I'm pushing on this is I think we should aim to make this interface as light and simple as possible for those trying to add price feeds since ultimately we'd like external devs to be able to do it.

I've considered extending PriceFeedInterface to create AggregatorPriceFeedInterface and BasePriceFeedInterface and every PriceFeed should implement one of the latter two. WDYT? Or is there a better way to address the core problem of this PR?

I think this is a good structure, primarily because it will keep the BasePriceFeeds as simple as possible. Most products will fall into the BasePriceFeeds category and hopefully, we can re-use the aggregator logic in different feed contexts. Once this logic is finalized we should document it very clearly to explain to price feed writers what the conditions/requirements are for each kind.

The other option would be to follow more of what @mrice32 is proposing wherein each feed itself can provide debugging information, either via returning multiple values (as proposed here, via a separate log debugger/helper utility or through a logger.

We should consider taking a step back and looking at the implications of both techniques and how they generalize given we are wanting to simplify the overall process of adding price feeds.

Great points. On one hand, if we were designing the PriceFeedInterface from scratch then I think returning two values [price, errorString] would make the most sense, where either price !== null || errorString !== null. This would allow the caller to check !price and then print out the errorString if the !price is true.

However implementing this change now would introduce a lot of churn into the code base because every caller (in the tests and clients) would now need to check for the [0]th element of getHistoricalPrice and there are a lot of such calls.

I think my PR introduces the least amount of churn but it does "muddy the water about what an implementer of the price feed class is expected to do". I think introducing two extensions of PriceFeedInterface could ease this concern: Base and Aggregator and we could use Base to improve documentation. Thoughts?

@nicholaspai nicholaspai requested a review from mrice32 January 22, 2021 12:30
Signed-off-by: Nick Pai <npai.nyc@gmail.com>
Signed-off-by: Nick Pai <npai.nyc@gmail.com>
Signed-off-by: Nick Pai <npai.nyc@gmail.com>
Signed-off-by: Nick Pai <npai.nyc@gmail.com>
@nicholaspai nicholaspai marked this pull request as ready for review January 31, 2021 22:42
@nicholaspai nicholaspai requested a review from mrice32 January 31, 2021 22:42
Signed-off-by: Nick Pai <npai.nyc@gmail.com>
Signed-off-by: Nick Pai <npai.nyc@gmail.com>
assert.equal(dexPriceFeed.getCurrentPrice(), null);
});

it("One event within window, several before", async function() {
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unrelated to the core of this PR, but I noticed that @mrice32 never added tests specific to #2465 . Do you agree that these tests correctly test changes introduced in that PR?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#2435 didn't affect the Balancer price feed but more tests are generally good!

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah these tests actually indicate the balancer price feed doesn't have the same problem that uniswap did (until your PR fixed it)!

assert.equal(uniswapPriceFeed.getCurrentPrice(), null);
});

it("One event within window, several before", async function() {
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same comment as above

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As mentioned here, the "Basic Historical TWAP" test actually ensures that time points before the window are taken into account in the TWAP -- otherwise the math wouldn't work (notice how the first time point is T-2.5 hours and the second time point is T-1.5 hours and the window starts at T-2 hours, so the first 30 minutes of the window is covered by the first time point). However, I think this test more explicitly tests that case!

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So is it correct to say that the latest data point before the window is used in the TWAP until the first data point within the window?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, that's correct.

Signed-off-by: Nick Pai <npai.nyc@gmail.com>
Signed-off-by: Nick Pai <npai.nyc@gmail.com>
Signed-off-by: Nick Pai <npai.nyc@gmail.com>
Signed-off-by: Nick Pai <npai.nyc@gmail.com>
Signed-off-by: Nick Pai <npai.nyc@gmail.com>
@nicholaspai
Copy link
Copy Markdown
Member Author

As mentioned IRL, after thinking about this more, I think keeping this interface simple and concise is important as it's becoming a focal point for external developers. I think this interface is already larger than it ideally should be, so I would prefer integrating the error messaging into the existing interface either by throwing or by returning objects of some type and having the error optionally added to those objects.
So one of the following would be my preferred approach:

return { price, error };

or

return [price, error];

or

assert(price, error);
return price;

Makes sense and I generally agree that creating two sub classes of PriceFeeds seems overkill for such a small feature. I think making the getXPrice methods return tuples will allow us to make clients such as the disputer more powerful in the future because they can better understand upstream pricefeed errors. Will plan to implement the getHistoricalPrice = (time) => return [price, errorString] interface.

@mrice32 @chrismaree @daywiss as discussed IRL, this PR changes the getHistoricalPrice interface so that it throws an error if a historical price is not returned for some reason.

Signed-off-by: Nick Pai <npai.nyc@gmail.com>
Signed-off-by: Nick Pai <npai.nyc@gmail.com>
Signed-off-by: Nick Pai <npai.nyc@gmail.com>
Signed-off-by: Nick Pai <npai.nyc@gmail.com>
Signed-off-by: Nick Pai <npai.nyc@gmail.com>
Copy link
Copy Markdown
Member

@mrice32 mrice32 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great! A few minor comments.

message: "Detected a disputable liquidation",
price: price.toString(),
liquidation: JSON.stringify(liquidation)
message: "Cannot dispute: price feed returned invalid value",
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Side note: @chrismaree @nicholaspai should we make this an error? (I know this was already a warn and it didn't change here). The reason I ask is because without that, this may not wake anyone up even though we're basically ignoring a liquidation because we can't price it.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I think it should be an error since the liquidator sends error level logs for failures to liquidate

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Made this an error level

if (time < this.lastUpdateTime - this.lookback) {
// Requesting an historical TWAP earlier than the lookback.
return null;
throw new Error(`${this.uuid} time ${time} is earlier than TWAP window`);
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These descriptions are going to be so nice.


if (this.computeMean) {
return this._computeMean(historicalPrices);
if (errors.length > 0 || historicalPrices.length === 0) {
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible for historicalPrices.length to be 0 here?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, good point.

assert.equal(dexPriceFeed.getCurrentPrice(), null);
});

it("One event within window, several before", async function() {
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#2435 didn't affect the Balancer price feed but more tests are generally good!

assert.equal(uniswapPriceFeed.getCurrentPrice(), null);
});

it("One event within window, several before", async function() {
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As mentioned here, the "Basic Historical TWAP" test actually ensures that time points before the window are taken into account in the TWAP -- otherwise the math wouldn't work (notice how the first time point is T-2.5 hours and the second time point is T-1.5 hours and the window starts at T-2 hours, so the first 30 minutes of the window is covered by the first time point). However, I think this test more explicitly tests that case!

Comment on lines +292 to +303
// If missing vol data, then the pricefeed is failing to return historical price data. Let's
// try catching the pricefeed's error to get more details about the problem:
let priceFeedErrorDetails;
try {
pricefeed.getHistoricalPrice(latestTime);
} catch (err) {
priceFeedErrorDetails = err.message;
}
return {
errorData: {
latestTime: latestTime ? latestTime : 0
latestTime: latestTime ? latestTime : 0,
priceFeedErrorDetails
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: thoughts on just getting _calculateHistoricalVolatility() to keep track of its last error and throwing it in the spot where it currently returns null?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm this is pretty interesting. So have _calculateHistoricalVolatility return successfully or throw, and have _checkPricrfeedVol report an error if it exists

Signed-off-by: Nick Pai <npai.nyc@gmail.com>
@nicholaspai nicholaspai requested a review from mrice32 February 2, 2021 03:39
Signed-off-by: Nick Pai <npai.nyc@gmail.com>
Copy link
Copy Markdown
Member

@mrice32 mrice32 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants