feat(financial-templates): adds a new price feed for the DefiPulseTotalTVL synthetic tokens#2346
feat(financial-templates): adds a new price feed for the DefiPulseTotalTVL synthetic tokens#2346nicholaspai merged 7 commits intoUMAprotocol:masterfrom bryanjcampbell1:master
Conversation
chrismaree
left a comment
There was a problem hiding this comment.
This looks like a great starting point! I've left some nits on formatting and input validation.
You should try to write some unit tests the same way we have unit tests for the cryptoWatchPriceFeed which you can see here: https://github.com/UMAprotocol/protocol/blob/master/packages/financial-templates-lib/test/price-feed/CryptoWatchPriceFeed.js
Also, to get this feed 100% integrated into this repo we will need an integration made to the CreatePriceFeed file. You can see the similar setup in this file for CryptoWatch here:
This will also contain some input validation using the isMissingField method, specifically for the apiKey.
| const { parseFixed } = require("@ethersproject/bignumber"); | ||
|
|
||
| // An implementation of PriceFeedInterface that uses DefiPulse Data api to retrieve prices. | ||
| class DEFI_PULSE_TOTAL_TVL_PriceFeed extends PriceFeedInterface { |
There was a problem hiding this comment.
nit: while the price feed is indeed called DEFI_PULSE_TOTAL_TVL This class should not be named like this. Ideally, it should be called DefiPulsePriceFeed I believe.
There was a problem hiding this comment.
Using DefiPulseTotalPriceFeed so I can differentiate between it and the SUSHI/UNI TVL price feed in the future
|
|
||
| let closestTime = {timestamp:0, value:0}; | ||
|
|
||
| //go through all values and find time that that is the largest and still less than 'time' |
There was a problem hiding this comment.
nit: here (and in other places) it does not seem you've linted your code correctly. We normally have spaces after // in the comments. You should try running the linter with yarn lint-fix to address these formatting issues.
There was a problem hiding this comment.
Is there a config file with a preferred linter settings somewhere in the repo? Im getting Command "lint-fix" not found when running yarn lint-fix
There was a problem hiding this comment.
did you install all the packages within the repo by running yarn in the root of the repo? it should work fine once you have done such.
There was a problem hiding this comment.
Got it. Thanks
| this.logger = logger; | ||
| this.web3 = web3; | ||
|
|
||
| this.apiKey = apiKey; |
There was a problem hiding this comment.
you might want to verify that there is indeed an API key added in here. If there is not, then this price feed will error out. This differs from the CryptoWatchPriceFeed as in those feeds the API key is optional whereas here it is a requirement.
You can simply do this with an assert(apiKey, "requires apiKey for Defi Pulse"); after importing const assert = require("assert");
| closestTime.value = val; | ||
|
|
||
| } | ||
|
|
There was a problem hiding this comment.
similar nit about formatting. Should be resolved once you run the linter.
| `https://data-api.defipulse.com/api/v1/defipulse/api/MarketData?api-key=${this.apiKey}`; | ||
|
|
||
| // 2. Send requests. | ||
| const [priceResponse] = await Promise.all([ |
There was a problem hiding this comment.
nit: no point in using Promise.all synthax here seeing you are only sending out one async call.
mrice32
left a comment
There was a problem hiding this comment.
This looks awesome! One comment about caching of historical data. It would also be great to see a few tests added to verify the behavior of this class!
Thanks for working on this!
|
|
||
| // 1. Construct URLs. | ||
| const priceUrl = | ||
| `https://data-api.defipulse.com/api/v1/defipulse/api/MarketData?api-key=${this.apiKey}`; |
There was a problem hiding this comment.
Is there any way to get historical data from defi pulse? It seems as if this class is depending on caching historical data over time internally, but generally the bots expect to be able to immediately access historical data up to the lookback. Does DefiPulse offer any API for immediate historical data?
There was a problem hiding this comment.
There is a GetHistory endpoint. How should I go about using it? Call it from getCurrentPrice() and store it in state so it can be referenced in getHistoricalPrice(time)?
There was a problem hiding this comment.
Modifying the CryptoWatchPriceFeed tests now. I looked more into the historical endpoint but the timestamps are a full day apart. I imagine the bots need more precision than this. Also, are there preferred ranges for lookback period and minTimeBetweenUpdates? Defi pulse only updates their api hourly.
| config.decimals, // This defaults to 18 unless supplied by user | ||
| config.ohlcPeriod // This defaults to 60 unless supplied by user | ||
| ); | ||
| } else if (config.type === "uniswap") { |
There was a problem hiding this comment.
why did you remove the existing uniswap feed? you should rather add your feed as another option rather than removing an existing feed that is used in production.
|
|
||
| } | ||
|
|
||
|
|
There was a problem hiding this comment.
this file still needs to be linted
| getTime, | ||
| minTimeBetweenUpdates | ||
| ); | ||
|
|
There was a problem hiding this comment.
this file also needs to be linted.
|
|
||
| const apiKey = process.env.DEFIPULSE_API_KEY; | ||
|
|
||
| const lookback = 3600*24; //??????????????? |
There was a problem hiding this comment.
what's up with this comment?
There was a problem hiding this comment.
This was a note to myself that I was unclear about an appropriate lookback time. The API only changes at hourly intervals so the bots would run less frequently than others like cryptowatch.
|
I made significant changes to the way the price feed the historical record. I am no longer building it by caching as I was able to get hourly values from the api. I am a little confused about the unit test though. How should I structure the fake data to inject? Also, how do I run the test? I tried calling 'truffle test' in root but just got a '0 passing' result. |
looks like you got the linter working, which is great! To run your test file you should navigate to the seeing you also changed the create price feed you should also test: I'll get your tests to run in CI now as well. |
|
Got the unit tests working! Do I need to worry about the git signoff errors? Running "git rebase HEAD~11 --signoff" returns "unknown option `signoff'" |
mrice32
left a comment
There was a problem hiding this comment.
Looks awesome -- a few minor comments, but I think this is just about ready!!
| } | ||
| } | ||
|
|
||
| const newPrice = this.web3.utils.toWei((mostRecent.tvlUSD / 1000000000).toFixed(6), "ether"); |
There was a problem hiding this comment.
Can you provide a comment on what's happening here?
Also, do you think this should be a class function since you do the same operation in the getHistoricalPrice method?
| } | ||
| } | ||
|
|
||
| const historicalPrice = this.web3.utils.toWei((closestTime.tvlUSD / 1000000000).toFixed(6), "ether"); |
There was a problem hiding this comment.
Does this respect the decimals value that is passed in the constructor?
There was a problem hiding this comment.
I removed decimals from the constructor. The UMIP does specify 3 decimals of precision though so I made that change. What I want is that when tvlUSD = 24781234567, the token price to be $24.781 or 24781000000000000000 wei of the collateral token.
There was a problem hiding this comment.
Am I understanding the price feed correctly? It should be returning a BN in wei correct?
| } | ||
| } | ||
|
|
||
| const historicalPrice = this.web3.utils.toWei((closestTime.tvlUSD / 1000000000).toFixed(3), "ether"); |
There was a problem hiding this comment.
Seeing you do this computation two times (this.web3.utils.toWei((closestTime.tvlUSD / 1000000000).toFixed(3), "ether");) you could consider refactoring this into a helper method within this feed that is called in two places. This way you could add a more detailed comment around the umip definition and help with code re-use.
There was a problem hiding this comment.
Also, you should consider adding this.toWei within the constructor, the same way you pulled out this.toBN on line 31. Then, you wont need to call this.web3.utils.toWei(... and can rather refer to this.toWei
you don't need to worry about the signoff part, we can set this from our side :) Also looks like you need to lint again! normally we have a husky hook that should lint your code automatically whenever you commit. If you using something other than the console to commit it's possible this is not enabled. I will re-run your tests through CI as well. |
|
What should my next steps be? I have run the cryptowatch bots on google cloud compute but I'm not sure how to configure to use my price feed. |
|
@bryanjcampbell1 sorry that I'm just coming back to this PR. Side note/quick tip: once reviewers send a round of comments and you've responded and would like them to take another look, you can click the cycle button next to their names (see screenshot below) to request them again! This basically adds it back to each of our personal review queues so we see and come back to the PR. See our contributing guidelines for more info on our typical process: https://github.com/UMAprotocol/protocol/blob/master/CONTRIBUTING.md#prs. I've gone ahead an re-requested myself and @chrismaree. Sorry for the delay!! |
|
@bryanjcampbell1 @chrismaree Actually, you do need to signoff your commits. We need outside contributors to complete the DCO for legal reasons. It's how we show that you really agree to submit your code under the repo license and forgo your individual rights to the code. |
| // Return early if the last call was too recent. | ||
| if (this.lastUpdateTime !== undefined && this.lastUpdateTime + this.minTimeBetweenUpdates > currentTime) { | ||
| this.logger.debug({ | ||
| at: "DEFI_PULSE_TOTAL_TVL_PriceFeed", |
There was a problem hiding this comment.
nit: you should make this at the same as the file name. I know this feed was previous all upper case like this but that's no longer the case.
| at: "DEFI_PULSE_TOTAL_TVL_PriceFeed", | |
| at: "DefiPulseTotalPriceFeed", |
| } | ||
|
|
||
| this.logger.debug({ | ||
| at: "DEFI_PULSE_TOTAL_TVL_PriceFeed", |
There was a problem hiding this comment.
| at: "DEFI_PULSE_TOTAL_TVL_PriceFeed", | |
| at: "DefiPulseTotalPriceFeed", |
| @@ -4677,6 +4677,15 @@ | |||
| "@uma/common" "^1.1.0" | |||
There was a problem hiding this comment.
not sure why there is a diff here?
| config.invertPrice // Not checked in config because this parameter just defaults to false. | ||
| config.invertPrice, // Not checked in config because this parameter just defaults to false. | ||
| config.poolDecimals, | ||
| config.priceFeedDecimals // This defaults to 18 unless supplied by user |
There was a problem hiding this comment.
why is this showing up as a diff? I'm pretty sure this is what the actual UniswapPriceFeed looks like.
There was a problem hiding this comment.
My guess is that there are some merge issues, where these changes accidentally got removed in the merge process, but then were re-added in this branch to get the tests to pass.
|
This looks really good to me and super close to being done! note that you still need to add in an additional set of changes in the Also, what is the intended collateral for this synthetic? importantly, if it has non 18 decimals you will also need to add it to This ensures that the decimals in the price requests returned by the dvm is scalled according to the collateral type defined for the specific synthetic. |
| let mockTime = 1611583300; | ||
| let networker; | ||
|
|
||
| const apiKey = process.env.DEFIPULSE_API_KEY; |
There was a problem hiding this comment.
unit tests should not rely on having this env set.
There was a problem hiding this comment.
Changed to a test string
|
@bryanjcampbell1 I updated the title to match the title in your description, just FYI! |
|
@bryanjcampbell1 Responding to this comment above (posting here because the comment is way up there and now outdated because you've made some updates :)): #2346 (comment):
|
| // In an effort to make the token price affordable, the value of the token is the tvlUSD divided by 1 billion. | ||
| // We also cut off precision after 3 decimals to match the specified price step of .001 | ||
|
|
||
| return this.toWei((_tvlUSD / 1000000000).toFixed(3), "ether"); |
There was a problem hiding this comment.
nit for readability:
| return this.toWei((_tvlUSD / 1000000000).toFixed(3), "ether"); | |
| const billion = 1_000_000_000; | |
| return this.toWei((_tvlUSD / billion).toFixed(3), "ether"); |
There was a problem hiding this comment.
Do most synths like this use USDC or TUSD? I had originally thought of using USDC (6 decimals).
There was a problem hiding this comment.
Any tips for signing off on old commits? Just set up a GPG keypair. Should be good signing off from here on.
|
What should I be using for my 'type' instead of 'medianizer'? I only have one source, the defipulse api, to get my value. |
Thanks! This was a huge help! |
mrice32
left a comment
There was a problem hiding this comment.
This is great! I left a few suggestions to make price feed decimals match how it works in the other price feeds. Please take a look. I haven't tested the suggestions, but I think they should work. You'll probably need to re-lint the repo, though, because they probably aren't formatted correctly. You can do this by committing the suggestions, running git pull, then running yarn lint-fix from the root of the repo, and then committing and pushing any changes.
packages/financial-templates-lib/src/price-feed/DefiPulseTotalPriceFeed.js
Outdated
Show resolved
Hide resolved
packages/financial-templates-lib/src/price-feed/DefiPulseTotalPriceFeed.js
Show resolved
Hide resolved
packages/financial-templates-lib/test/price-feed/DefiPulseTotalPriceFeed.js
Show resolved
Hide resolved
packages/financial-templates-lib/test/price-feed/DefiPulseTotalPriceFeed.js
Show resolved
Hide resolved
| networker, | ||
| getTime, | ||
| minTimeBetweenUpdates, | ||
| 6 // Add arbitrary decimal conversion and prove this works. |
There was a problem hiding this comment.
| 6 // Add arbitrary decimal conversion and prove this works. | |
| decimals // Add arbitrary decimal conversion and prove this works. |
| assert.equal((await defiPulseTotalPriceFeed.getHistoricalPrice(1611575900)).toString(), web3.utils.toWei("24.78")); | ||
| assert.equal((await defiPulseTotalPriceFeed.getHistoricalPrice(1611579500)).toString(), web3.utils.toWei("23.50")); | ||
| assert.equal((await defiPulseTotalPriceFeed.getHistoricalPrice(1611579500)).toString(), web3.utils.toWei("23.50")); | ||
| assert.equal((await defiPulseTotalPriceFeed.getHistoricalPrice(1611579601)).toString(), web3.utils.toWei("22.25")); |
There was a problem hiding this comment.
Since the decimals are specified as 6, shouldn't these be coming out as 24.78 * 10^6? Right now, it looks like they're coming out as 24.78 * 10^18. See my comment above for how to make the price feed to do what you expect here.
| assert.equal((await defiPulseTotalPriceFeed.getHistoricalPrice(1611575900)).toString(), web3.utils.toWei("24.78")); | |
| assert.equal((await defiPulseTotalPriceFeed.getHistoricalPrice(1611579500)).toString(), web3.utils.toWei("23.50")); | |
| assert.equal((await defiPulseTotalPriceFeed.getHistoricalPrice(1611579500)).toString(), web3.utils.toWei("23.50")); | |
| assert.equal((await defiPulseTotalPriceFeed.getHistoricalPrice(1611579601)).toString(), web3.utils.toWei("22.25")); | |
| assert.equal((await defiPulseTotalPriceFeed.getHistoricalPrice(1611575900)).toString(), parseFixed("24.78", decimals).toString()); | |
| assert.equal((await defiPulseTotalPriceFeed.getHistoricalPrice(1611579500)).toString(), parseFixed("23.50", decimals).toString()); | |
| assert.equal((await defiPulseTotalPriceFeed.getHistoricalPrice(1611579500)).toString(), parseFixed("23.50", decimals).toString()); | |
| assert.equal((await defiPulseTotalPriceFeed.getHistoricalPrice(1611579601)).toString(), parseFixed("22.25", decimals).toString()); |
| await defiPulseTotalPriceFeed.update(); | ||
|
|
||
| // Should return the current price in the data. | ||
| assert.equal(defiPulseTotalPriceFeed.getCurrentPrice().toString(), web3.utils.toWei("25.1")); |
There was a problem hiding this comment.
| assert.equal(defiPulseTotalPriceFeed.getCurrentPrice().toString(), web3.utils.toWei("25.1")); | |
| assert.equal(defiPulseTotalPriceFeed.getCurrentPrice().toString(), parseFixed("25.1", decimals).toString()); |
|
Accepted changes and ran through linter! |
mrice32
left a comment
There was a problem hiding this comment.
I made a few small commits because I didn't realize you had manually made some of the changes I suggested -- then I had to undo them haha. But I think it looks good now!
LGTM assuming @chrismaree approves as well!
packages/financial-templates-lib/src/price-feed/DefiPulseTotalPriceFeed.js
Show resolved
Hide resolved
packages/financial-templates-lib/src/price-feed/DefiPulseTotalPriceFeed.js
Show resolved
Hide resolved
chrismaree
left a comment
There was a problem hiding this comment.
This is so close to being ready. I have 2 nits and questions on some small spots.
Also, please add a test to validate non-18 decimals works as expected. Once this is done I think you good to go!
| networker, | ||
| getTime, | ||
| config.minTimeBetweenUpdates, | ||
| config.decimals |
There was a problem hiding this comment.
nit: for consistency please keep this as
| config.decimals | |
| config.priceFeedDecimals |
| assert.equal(defiPulseTotalPriceFeed.getLastUpdateTime(), undefined); | ||
| }); | ||
|
|
||
| it("Basic historical price", async function() { |
There was a problem hiding this comment.
ye, please add a test for this :)
…alTVL synthetic tokens Signed-off-by: bryanjcampbell1 <bryanjcampbell1@gmail.com>
|
@chrismaree Matt helped me modify the test for "Basic historical price" so that it would work for arbitrary decimals. Please let me know if you are looking for something else! |
Signed-off-by: bryanjcampbell1 <bryanjcampbell1@gmail.com>
chrismaree
left a comment
There was a problem hiding this comment.
Thanks for replying to all the changes and updating your PR accordingly! this is pretty awesome, I'm looking forward to seeing this on main net 🚀
|
I'm running your test in CI again. |
| } else if (config.type === "defipulsetvl") { | ||
| const requiredFields = ["lookback", "minTimeBetweenUpdates"]; | ||
|
|
||
| if (isMissingField(config, config.apiKey, requiredFields, logger)) { |
There was a problem hiding this comment.
Why don't you add the apiKey to requiredFields?
|
|
||
| async getHistoricalPrice(time) { | ||
| if (this.lastUpdateTime === undefined) { | ||
| throw new Error(`${this.uuid}: undefined lastUpdateTime`); |
There was a problem hiding this comment.
I don't see that you've defined this.uuid in the constructor
| this.web3 = web3; | ||
|
|
||
| this.apiKey = apiKey; | ||
| assert(apiKey, "requires apiKey for Defi Pulse"); |
There was a problem hiding this comment.
Instead of asserting it here, I recommend adding apiKey to the requiredFields array in CreatePriceFeed
nicholaspai
left a comment
There was a problem hiding this comment.
I left some comments that once resolved are good to merge
Signed-off-by: bryanjcampbell1 <bryanjcampbell1@gmail.com>
Merge branch 'master' of https://github.com/bryanjcampbell1/protocol into bryanjcampbell1/master
|
|
||
| async getHistoricalPrice(time) { | ||
| if (this.lastUpdateTime === undefined) { | ||
| throw new Error("undefined lastUpdateTime"); |
There was a problem hiding this comment.
Can you follow this this.uuid pattern and add a uuid class param that makes the Error slightly more descriptive please? Sorry and thanks!
|
@bryanjcampbell1 looks like there is just one outstanding issue and this can go in! |


Title
feat(financial-templates): adds a new price feed for the DefiPulseTotalTVL synthetic tokens
Motivation
UMIP 24 introduces DefiPulseTotalTVL as a price id. The price feed suggested in this PR would allow for the creation of liquidation and dispute bots for the DefiPulseTotalTVL price id.
Summary
Adds the DefiPulseTotalTVLPriceFeed.js file to the financial-templates repo
Details
This price feed was modified from, and closely follows, the CryptoWatchPriceFeed. The cryptowatch api has been replaced with the DeFi Pulse Data API.
Issue(s)
Fixes the current lack of a price feed for the DefiPulseTotalTVL price id.