Skip to content

fix: coverage for libraries#7510

Merged
klkvr merged 10 commits into
masterfrom
klkvr/coverage-libraries
Mar 29, 2024
Merged

fix: coverage for libraries#7510
klkvr merged 10 commits into
masterfrom
klkvr/coverage-libraries

Conversation

@klkvr

@klkvr klkvr commented Mar 28, 2024

Copy link
Copy Markdown
Member

Motivation

Closes #7054
Closes #2567
Closes #4854
Closes #4654

Solution

Currently we're collecting anchors via the following steps:

  1. Walk the AST for each contract definition to find coverage items contained in it along with its base contracts and potentially libraries.
  2. Collect mapping from contract_id to coverage items contained in it itself or its base contracts/libraries.
  3. Try to find anchors for each coverage item found in step 2.

This approach requires us to be able to reliably find libraries used by a contract which turned out to be non-trivial.

One possible approach could be to reuse some of #6936 code to walk through typed AST and find all contracts/libs used by a given contract. However, this might be breaking due to typed AST not working for some solc versions and this approach is also not perfect.

Instead, we now just collect all coverage items and then create a mapping from source_id to all coverage items given source contains.

After that, when looking for anchors, we only try finding anchors for items from source ids appearing in source map. This allows us to get rid of dependency resolution through AST.

Example

I was mainly testing those on Uniswap/v4-core contracts.

Report via latest nightly: report_nightly.txt
Report via this PR: report_pr.txt

diff:

9,11c9,11
< | src/libraries/BitMath.sol                  | 0.00% (0/47)      | 0.00% (0/57)       | 0.00% (0/36)     | 0.00% (0/2)      |
< | src/libraries/FullMath.sol                 | 0.00% (0/29)      | 0.00% (0/33)       | 0.00% (0/8)      | 0.00% (0/2)      |
< | src/libraries/Hooks.sol                    | 0.00% (0/41)      | 0.00% (0/76)       | 0.00% (0/28)     | 0.00% (0/14)     |
---
> | src/libraries/BitMath.sol                  | 100.00% (47/47)   | 100.00% (57/57)    | 100.00% (36/36)  | 100.00% (2/2)    |
> | src/libraries/FullMath.sol                 | 100.00% (29/29)   | 100.00% (33/33)    | 100.00% (8/8)    | 100.00% (2/2)    |
> | src/libraries/Hooks.sol                    | 100.00% (41/41)   | 98.68% (75/76)     | 96.43% (27/28)   | 100.00% (14/14)  |
14,23c14,23
< | src/libraries/Pool.sol                     | 0.00% (0/130)     | 0.00% (0/141)      | 0.00% (0/86)     | 0.00% (0/13)     |
< | src/libraries/PoolGetters.sol              | 0.00% (0/2)       | 0.00% (0/2)        | 100.00% (0/0)    | 0.00% (0/2)      |
< | src/libraries/Position.sol                 | 0.00% (0/12)      | 0.00% (0/14)       | 0.00% (0/6)      | 0.00% (0/2)      |
< | src/libraries/SafeCast.sol                 | 0.00% (0/8)       | 0.00% (0/14)       | 0.00% (0/8)      | 0.00% (0/4)      |
< | src/libraries/SqrtPriceMath.sol            | 0.00% (0/32)      | 0.00% (0/66)       | 0.00% (0/24)     | 0.00% (0/8)      |
< | src/libraries/SwapFeeLibrary.sol           | 0.00% (0/5)       | 0.00% (0/9)        | 0.00% (0/4)      | 0.00% (0/3)      |
< | src/libraries/SwapMath.sol                 | 0.00% (0/23)      | 0.00% (0/30)       | 0.00% (0/12)     | 0.00% (0/1)      |
< | src/libraries/TickBitmap.sol               | 0.00% (0/19)      | 0.00% (0/34)       | 0.00% (0/6)      | 0.00% (0/3)      |
< | src/libraries/TickMath.sol                 | 97.87% (92/94)    | 92.57% (137/148)   | 82.61% (38/46)   | 50.00% (2/4)     |
< | src/libraries/UnsafeMath.sol               | 0.00% (0/1)       | 0.00% (0/1)        | 100.00% (0/0)    | 0.00% (0/1)      |
---
> | src/libraries/Pool.sol                     | 96.92% (126/130)  | 94.33% (133/141)   | 89.53% (77/86)   | 100.00% (13/13)  |
> | src/libraries/PoolGetters.sol              | 100.00% (2/2)     | 100.00% (2/2)      | 100.00% (0/0)    | 100.00% (2/2)    |
> | src/libraries/Position.sol                 | 100.00% (12/12)   | 100.00% (14/14)    | 100.00% (6/6)    | 100.00% (2/2)    |
> | src/libraries/SafeCast.sol                 | 100.00% (8/8)     | 100.00% (14/14)    | 100.00% (8/8)    | 100.00% (4/4)    |
> | src/libraries/SqrtPriceMath.sol            | 100.00% (32/32)   | 98.48% (65/66)     | 83.33% (20/24)   | 100.00% (8/8)    |
> | src/libraries/SwapFeeLibrary.sol           | 100.00% (5/5)     | 100.00% (9/9)      | 100.00% (4/4)    | 100.00% (3/3)    |
> | src/libraries/SwapMath.sol                 | 100.00% (23/23)   | 100.00% (30/30)    | 100.00% (12/12)  | 100.00% (1/1)    |
> | src/libraries/TickBitmap.sol               | 100.00% (19/19)   | 100.00% (34/34)    | 100.00% (6/6)    | 100.00% (3/3)    |
> | src/libraries/TickMath.sol                 | 97.87% (92/94)    | 97.30% (144/148)   | 97.83% (45/46)   | 50.00% (2/4)     |
> | src/libraries/UnsafeMath.sol               | 100.00% (1/1)     | 100.00% (1/1)      | 100.00% (0/0)    | 100.00% (1/1)    |
51,53c51,53
< | src/types/BalanceDelta.sol                 | 0.00% (0/2)       | 0.00% (0/2)        | 100.00% (0/0)    | 0.00% (0/2)      |
< | src/types/Currency.sol                     | 0.00% (0/15)      | 0.00% (0/24)       | 0.00% (0/10)     | 0.00% (0/6)      |
< | src/types/PoolId.sol                       | 0.00% (0/1)       | 0.00% (0/2)        | 100.00% (0/0)    | 0.00% (0/1)      |
---
> | src/types/BalanceDelta.sol                 | 100.00% (2/2)     | 100.00% (2/2)      | 100.00% (0/0)    | 100.00% (2/2)    |
> | src/types/Currency.sol                     | 100.00% (15/15)   | 91.67% (22/24)     | 80.00% (8/10)    | 100.00% (6/6)    |
> | src/types/PoolId.sol                       | 100.00% (1/1)     | 100.00% (2/2)      | 100.00% (0/0)    | 100.00% (1/1)    |
55c55
< | test/utils/Deployers.sol                   | 0.00% (0/50)      | 0.00% (0/63)       | 0.00% (0/10)     | 0.00% (0/11)     |
---
> | test/utils/Deployers.sol                   | 96.00% (48/50)    | 93.65% (59/63)     | 80.00% (8/10)    | 90.91% (10/11)   |
58,59c58,59
< | test/utils/SortTokens.sol                  | 0.00% (0/3)       | 0.00% (0/5)        | 0.00% (0/2)      | 0.00% (0/1)      |
< | Total                                      | 43.42% (554/1276) | 42.89% (745/1737)  | 22.24% (153/688) | 40.35% (115/285) |
---
> | test/utils/SortTokens.sol                  | 66.67% (2/3)      | 80.00% (4/5)       | 50.00% (1/2)     | 100.00% (1/1)    |
> | Total                                      | 75.78% (967/1276) | 75.30% (1308/1737) | 55.38% (381/688) | 66.67% (190/285) |

cc @onbjerg

@klkvr klkvr changed the title fix: coverage for external libraries fix: coverage for libraries Mar 28, 2024

@onbjerg onbjerg left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks great overall, really stoked, an initial q tho

// end up here again
if items.is_empty() || is_test {
self.contract_items.insert(contract_id.clone(), Vec::new());
let ContractVisitor { items, .. } = ContractVisitor::new(

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

w/o the if !self.contract_items.contains_key(contract_id) { check, will this not lead to re-analyzing contracts? and possibly duping coverage items?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We're walking only ContractDefinition nodes from self.contracts, so it should be fine as long as there is no duplicating ContractId keys

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think thats a fair assumption

@onbjerg onbjerg added T-bug Type: bug Cmd-forge-coverage Command: forge coverage labels Mar 28, 2024

@mattsse mattsse left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

love to see it.

only a few style nits, defer to @onbjerg

is there a few to add a few sanity tests for this? probably difficult?

Comment thread crates/evm/coverage/src/analysis.rs Outdated
Comment thread crates/forge/bin/cmd/coverage.rs Outdated

@mattsse mattsse left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm, pending @onbjerg

Comment thread crates/forge/bin/cmd/coverage.rs Outdated
.contract_items

// Build helper mapping used by `find_anchors`
let mut items_by_source_id = HashMap::new();

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
let mut items_by_source_id = HashMap::new();
let mut items_by_source_id = HashMap::with_capacity(source_analysis.items.len());

Comment thread crates/forge/bin/cmd/coverage.rs Outdated
let mut items_by_source_id = HashMap::new();

for (item_id, item) in source_analysis.items.iter().enumerate() {
items_by_source_id.entry(item.loc.source_id).or_insert_with(Vec::new).push(item_id);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there should be or_default() I believe

@klkvr klkvr merged commit 452956f into master Mar 29, 2024
@klkvr klkvr deleted the klkvr/coverage-libraries branch March 29, 2024 13:34
ZhangZhuoSJTU pushed a commit to MEDGA-eth/foundry that referenced this pull request Mar 29, 2024
* fix: coverage for internal libraries

* optimize

* optimize

* doc

* rm tests

* clippy

* clippy + fmt

* clean up

* for loop

* review fixes
@frontier159

Copy link
Copy Markdown
Contributor

This sounds great, thanks @klkvr

It seems to have a regression for different --report arguments though:

  1. If --report summary is explicitly used, it shows zero coverage for everything (should be the same as not specifying according to the --help)
  2. If --report lcov is used, it shows a lot less coverage than before the change

FWIW, I run via this flow to get a html representation of the coverage and branches:

forge coverage --report summary --report lcov
lcov --remove lcov.info 'contracts/test/*' 'test/*' --output-file lcov.info --rc lcov_branch_coverage=1
genhtml lcov.info -o report --branch-coverage --legend

@klkvr

klkvr commented Mar 31, 2024

Copy link
Copy Markdown
Member Author

@frontier159 Thank you for reporting and really sorry for breaking those. Any chance you could share a repository on which this can be reproduced?

@frontier159

frontier159 commented Mar 31, 2024

Copy link
Copy Markdown
Contributor

@klkvr it seems after a forge clean it works, but then running a second time gives near zero coverage.

You can try this repo:

  1. clone with submodules
  2. cd apps/protocol
  3. nvm use
  4. yarn
  5. yarn forge-coverage-html

After a yarn clean && forge clean gives the correct coverage. But then running yarn forge-coverage-html a second time (without a clean) gives almost zero coverage

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Cmd-forge-coverage Command: forge coverage T-bug Type: bug

Projects

None yet

4 participants