FLIP: Variable transaction fees #660

janezpodhostnik · 2021-10-07T15:58:52Z

Description

First draft

For contributor use:

Targeted PR against master branch
Linked to Github issue with discussion and accepted design OR link to spec that describes this work.
Updated relevant documentation
Re-reviewed Files changed in the Github PR explorer
Added appropriate labels

vercel · 2021-10-07T15:58:55Z

This pull request is being automatically deployed with Vercel (learn more).
To see the status of your deployment, click below or on the icon next to each commit.

🔍 Inspect: https://vercel.com/onflow/flow-docs/86CW2E8N7qan1cdeAft9PGENyoBj
✅ Preview: https://flow-docs-git-janez-transaction-fees-onflow.vercel.app

[Deployment for 2f48991 failed]

…n-fees

janezpodhostnik · 2021-11-23T19:09:01Z

Updated to simplify how the transaction fees are split up.

The new proposed way is to split them int a known part and unknown part (for the lack of better words). The only way to the unknown part is to actually run the transaction.

flips/20211007-transaction-fees.md

j1010001 · 2021-11-24T18:36:52Z

flips/20211007-transaction-fees.md

+3. A potential third use of the effort cost factor would be to keep the cost of transaction stable according to the USD value of flow. This would require a periodic job to run, check the USD value of FLOW and update the cost factor to account for any changes. This would allow for the average transaction fees to stay relatively close to a set USD value.

-While the effort cost factors could be adjusted manually from the start, that would mean responding to network load would be very slow. A potential solution to adjusting the effort cost factors automatically would be a service that monitored the networks performance and sent a transaction to update the fee factors when there was a change in the network load.
+While the effort cost factor could be adjusted manually from the start, that would mean responding to network load would be very slow. A potential solution to adjusting the effort cost factor automatically would be a service that monitored the networks performance and sent a transaction to update the fee factors when there was a change in the network load.


Do you mean network load instead of performance ?

A potential solution to adjusting the effort cost factor automatically would be a service that monitored the networks performance and sent a transaction to update the fee factors when there was a change in the network load

+1

That would be a viable proposal, if such service is universally trusted. But who would be running such service for a fully decentralized system? How to we guarantee correct functioning of such service in a byzantine environment?

j1010001 · 2021-11-24T18:42:26Z

flips/20211007-transaction-fees.md


-The constant inclusion effort could be chosen by observing the impact of transactions with a different byte size and a similar computation effort. This data could be used to predict how much impact a transaction that is 0 bytes long would make.
+1. Access node:
+    1. Sending a transaction causes load on the access node (might depend on the byte size of the transaction).


"The access node needs to potentially do a check (at a constant effort)" - Is this speculation on what future might bring, or does the access node perform some checks already ? If it does I'd just say what it is checking for, otherwise this will probably raise questions.

"The access node needs to potentially do a check (at a constant effort)" this statement is incomplete. My focus probably jumped here. The statement is supposed to be: "The access node needs to potentially do a check that the payer can pay (at a constant effort)". No this check does not exist, I will try to make that clearer.

This list is presented as a list of "potential factors" (stated in the paragraph above the list).

Does bandwidth refer to networking bandwidth, compute bandwidth, both, or neither?

j1010001 · 2021-11-24T19:30:51Z

flips/20211007-transaction-fees.md

+4. Transaction failed because the execution effort limit was reached.

-As discussed below, in scenario 1. the transaction shouldn't have been included in a collection in the first place. However since it was included and there is no way to charge the payer, the collection node account is charged with the transaction fees. The transaction fees are computed with the dynamic computation effort set to 0.
+As discussed below, in scenario 1. the transaction shouldn't have passed the access node in the first place. However since it has and there is no way to charge the payer, the access node account is charged with the transaction fees. The transaction fees are computed with the execution effort set to 0.


I'd probably merge the proposal on the charges with the cases above, otherwise I am jumping back and forth between the different cases and how we propose to charge for them.

I switched the order of the chapters, so at the point when you are reading about the categorisation of failing transactions, you should already have knowledge about some transactions not being included due to the payer being unable to pay.

Flagging that we should discuss whether or not this is managed at AN or LN (collection node) level. I think currently we're trending toward AN but I do think we should complete the discussion on how failing txs without payer are managed between those 2 node types. I know @AlexHentschel has many thoughts ;)

flips/20211007-transaction-fees.md

AlexHentschel

Thanks @janezpodhostnik for the detailed write up. Great, that really enables a systematic discussion around which pricing behaviour we desire.
While reading, I found that the mathematical model is a bit hard to follow at times. I think that is due to varying notation and terminology on the one hand and a somewhat unusual mathematical formulation on the other hand. While from a purely mathematical perspective, a variety of formulations are possible to describe the same behaviour, some formulations are more intuitive for as humans than others.
I have made detailed comments on where I would suggest to revise the mathematical model to better align with common intuitions and terms. Another point which deserves more discussion (and potentially refinement) is how we chose a suitable parameter setting to induce the desired client behaviour. I generally agree that some form of regression will be useful; though I don't quite understand the detailed proposition around "extrapolating for a 0 size transaction". I think we should probably have a follow-up discussion (in person) around this.

flips/20211007-transaction-fees.md

AlexHentschel · 2021-11-25T21:00:11Z

flips/20211007-transaction-fees.md

+From the protocols perspective the following requirements can be set:
+
+- Transactions that require more resources from the network to be processed, should cost more.
+- If the protocol is experiencing a lot of traffic, all transactions should become more expensive, until the high traffic subsides.


Suggested change

- If the protocol is experiencing a lot of traffic, all transactions should become more expensive, until the high traffic subsides.

- Surge pricing: If the protocol is experiencing a lot of traffic, all transactions should become more expensive, until the high traffic subsides.

Flagging (mostly to myself) that we should sanity check this notion

flips/20211007-transaction-fees.md

AlexHentschel · 2021-11-26T00:05:51Z

flips/20211007-transaction-fees.md

+
+3. A potential third use of the effort cost factor would be to keep the cost of transaction stable according to the USD value of flow. This would require a periodic job to run, check the USD value of FLOW and update the cost factor to account for any changes. This would allow for the average transaction fees to stay relatively close to a set USD value.
+
+While the effort cost factor could be adjusted manually from the start, that would mean responding to network load would be very slow. A potential solution to adjusting the effort cost factor automatically would be a service that monitored the networks performance and sent a transaction to update the fee factors when there was a change in the network load.


Suggested change

While the effort cost factor could be adjusted manually from the start, that would mean responding to network load would be very slow. A potential solution to adjusting the effort cost factor automatically would be a service that monitored the networks performance and sent a transaction to update the fee factors when there was a change in the network load.

While the effort cost could be adjusted manually from the start, that would mean responding to network load would be very slow. A potential solution to adjusting the effort cost factor automatically would be a service that monitored the networks performance and sent a transaction to update the fee factors when there was a change in the network load.

AlexHentschel · 2021-11-26T00:25:16Z

flips/20211007-transaction-fees.md

+For the first iteration the inclusion effort could be defined to be relative to the byte length of the (rlp encoded) transaction (scaled to match average execution effort) plus a well chosen constant part. The constant inclusion effort part could be chosen by observing the impact of transactions with a different byte size and a similar execution effort. This data could be used to predict how much impact a transaction that is 0 bytes long would make.
+
+Having a positive constant inclusion effort as well as a dynamic one means that splitting a transaction into smaller pieces does not make sense (considering only the inclusion part of the transaction fees). If this would be desired (from the protocols perspective) that the transactions are as small as possible or of a certain size, the dynamic inclusion effort could be defined to grow faster then linear with the byte size of the transaction.
+


I would consider it likely that for the lower effort range, you computation will likely roughly increase linearly. But there is also the range of "excessive usage", which we want to disincentivize. A simple example for a curve combining both could be:

AlexHentschel · 2021-11-26T01:02:08Z

flips/20211007-transaction-fees.md

+3. A potential third use of the effort cost factor would be to keep the cost of transaction stable according to the USD value of flow. This would require a periodic job to run, check the USD value of FLOW and update the cost factor to account for any changes. This would allow for the average transaction fees to stay relatively close to a set USD value.
+
+While the effort cost factor could be adjusted manually from the start, that would mean responding to network load would be very slow. A potential solution to adjusting the effort cost factor automatically would be a service that monitored the networks performance and sent a transaction to update the fee factors when there was a change in the network load.
+


I have another question around 'effort' vs 'effort cost'. Usually, in economy There is a certain unit of work (in our terminology effort), but depending on demand, there would be a different price (here 'effort cost').

I would consider it likely that for the lower effort range, our cost per unit of work scales roughly linearly. But there is also the range of "excessive usage", which we want to disincentivize. A simple example for a curve combining both could be:

The specific point I am tying to get here is that Effort usually describes some resource consumption (e.g. consumed CPU cycles). The variable that represents "the cost of expending that effort" in this model here is f (the price).

For for example, below you talk about the Inclusion effort. For a transaction, it consumes a certain amount of effort (CPU cycles and networking bandwidth). The parameter, by which I incentivize or disincentivize is the price. Since you placed this discussion in the section describing an effort, it sounds like you want to change the definition effort but in fact, we are controlling the price, i.e. 'effort cost'.
In your sections on effort, you mention efforts like networking bandwidth or CPU cycles. Describing such economic systems, there is generally an individual effort factor assigned to each.
I would suggest to adjust the mathematical model to:

F = F_i + F_x = f ( p_i * E_i + p_x * E_x) where * `f` is the surge factor to control the overall network load * `E_i` is the inclusion effort, this is a measure of physical resource consumption * `p_i` is the effort cost, i.e. price. The price is generally sensitive to the _amount_ of resources expended. Mathematically this means that `p_i` is a _function of `E_i` and we write `p_i(E_i)`. For illustration see the figure above.

I think this makes sense, but would this apply to inclusion effort and execution effort, or just execution?

AlexHentschel · 2021-11-26T04:31:06Z

flips/20211007-transaction-fees.md

+- Is the payers signature valid.
+- Can the payer pay for the transaction. The payers default FLOW vault should have at least <img src="https://render.githubusercontent.com/render/math?math=F%5E%5Ctext%7Bmax%7D"> FLOW. <img src="https://render.githubusercontent.com/render/math?math=F%5E%7Bmax%7D%20%3D%20f%20(%20E_i%20%2B%20E_e%5E%7Bmax%7D%20)">
+
+It might still happen that this transaction will get included in a collection, either due to a bug or malicious behaviour from the access node, or due to the access node not having up to date information of the balance of the payer. In case does still get included in a collection, even though it does not meet the condition, this check should be repeated in the execution node, and in case the check fails on the execution node, the transaction fees should be deducted from the access nodes account instead of the payers account.


Wording suggestion:

Suggested change

It might still happen that this transaction will get included in a collection, either due to a bug or malicious behaviour from the access node, or due to the access node not having up to date information of the balance of the payer. In case does still get included in a collection, even though it does not meet the condition, this check should be repeated in the execution node, and in case the check fails on the execution node, the transaction fees should be deducted from the access nodes account instead of the payers account.

It might still happen that a transaction violating these criteria is included in a collection, either due to a bug or malicious behaviour from the access node, or due to the access node not having up to date information of the balance of the payer. Therefore, the same checks should be repeated by the execution nodes, and in case the transactions fails these specific criteria, the transaction fees is deducted from the access node's account instead of the payer's.

AlexHentschel · 2021-11-26T05:36:39Z

flips/20211007-transaction-fees.md

+- <img src="https://render.githubusercontent.com/render/math?math=E_i"> Inclusion effort would be the byte size of a transaction plus a constant inclusion effort that would be picked by using linear regression on the impact of different size transactions (with the same execution effort) on the network, and extrapolating for a 0 size transaction.
+- <img src="https://render.githubusercontent.com/render/math?math=f_i">: Effort cost factor would be picked so that for 95% of transactions that have been seen so far the inclusion fees would cost less then `TC/2`.


regarding parameter calibration:

so we have effort, e.g. we could define that every 1kb transaction size equals 1 unit of inclusion effort

and then we have a effort cost, i.e. how much does 1kb transaction size costs (?)

In economy, you often draw a price vs quantity diagram:

I have introduced several quantities here:

E_95 the 95% effort percentile of current transactions (i.e. their size in kb)

F_c the current (fixed) transaction fee

Furthermore, lets consider another transaction A with different effort (This could be a very small transaction with close to minimal effort E_min, as shown above) or a larger transactions (illustrated below).

Assuming you desire a linear cost-effort relationship:

this is a linear function which is uniquely defined by a single point X and a slope α

we already have one point fixed: X_95 = (E_95, F_c)

for another transaction with effort E_A we still lack a meaningful price value F_A

So how do we determine F_A?

This is where regression comes into play, but I don't think the ordinate intersection point is particularly helpful here (you call this the "0-size transaction"). As an illustration, consider the two different price curves p^(1) vs p^(2) in the figure above. While they differ in their slope, i.e. in how much they charge per unit of effort (price per 1kb tx size), they are both reasonable pricing models.

I think for deciding which curve to pick, we should look at the throughput limitations for some sample transactions. Roughly speaking, if we have two transactions A and B and we get twice as many transactions of type A through the system compared to transaction B, transaction B should cost twice as much.

Formally, we require at least two benchmark transactions, A and B with their respective size E_A and E_B

We could load our benchmarking system with only these transactions up to the limit and measure the maximally achievable throughput. By comparing the throughput limitations, we can infer the slope. If we use more than 2 transactions, we get a sample of possible slopes.

flips/20211007-transaction-fees.md

laynelafrance · 2021-12-02T02:57:22Z

flips/20211007-transaction-fees.md


-The constant inclusion effort could be chosen by observing the impact of transactions with a different byte size and a similar computation effort. This data could be used to predict how much impact a transaction that is 0 bytes long would make.
+1. Access node:
+    1. Sending a transaction causes load on the access node (might depend on the byte size of the transaction).


Does bandwidth refer to networking bandwidth, compute bandwidth, both, or neither?

laynelafrance · 2021-12-02T02:58:20Z

flips/20211007-transaction-fees.md

+From the protocols perspective the following requirements can be set:
+
+- Transactions that require more resources from the network to be processed, should cost more.
+- If the protocol is experiencing a lot of traffic, all transactions should become more expensive, until the high traffic subsides.


Flagging (mostly to myself) that we should sanity check this notion

laynelafrance · 2021-12-02T02:59:04Z

flips/20211007-transaction-fees.md

+- Transactions that require more resources from the network to be processed, should cost more.
+- If the protocol is experiencing a lot of traffic, all transactions should become more expensive, until the high traffic subsides.
+
+The ultimate goal from the protocols perspective would be that transactions cost exactly the amount of FLOW needed to pay for the cost of the time needed by the nodes to process the transaction. However this would include so many variables that it would not be feasible. Instead the goal is to make a reasonably good approximation, and improve that approximation over time.


+1 We should explain that what’s expensive is the cost of “looking and running”

Does bandwidth refer to networking bandwidth, compute bandwidth, both, or neither?

Both, but primarily networking bandwidth. Not sure its worth elaborating on this here, as this is mostly a list of potential factors...

laynelafrance · 2021-12-02T03:00:48Z

flips/20211007-transaction-fees.md

+
+This separation into a part that can be know before the transaction is executed, and a part that cannot be know before the transaction is executed, allows for better oversight from the user into how much the transaction will actually cost.
+
+Each part of the fees can be further split into *effort*(<img src="https://render.githubusercontent.com/render/math?math=E">) and an *effort cost factor*(<img src="https://render.githubusercontent.com/render/math?math=f">) of FLOW per unit of effort. The *effort* required for a specific transaction should be independent of when the transaction is sent[^1] and should only depend on the properties of the transaction. On the other hand the *effort cost factor* would change over time depending on network activity.


"effort" is defined as cost of looking at the tx and running it?

flips/20211007-transaction-fees.md

laynelafrance · 2021-12-02T03:03:21Z

flips/20211007-transaction-fees.md

+3. A potential third use of the effort cost factor would be to keep the cost of transaction stable according to the USD value of flow. This would require a periodic job to run, check the USD value of FLOW and update the cost factor to account for any changes. This would allow for the average transaction fees to stay relatively close to a set USD value.

-While the effort cost factors could be adjusted manually from the start, that would mean responding to network load would be very slow. A potential solution to adjusting the effort cost factors automatically would be a service that monitored the networks performance and sent a transaction to update the fee factors when there was a change in the network load.
+While the effort cost factor could be adjusted manually from the start, that would mean responding to network load would be very slow. A potential solution to adjusting the effort cost factor automatically would be a service that monitored the networks performance and sent a transaction to update the fee factors when there was a change in the network load.


A potential solution to adjusting the effort cost factor automatically would be a service that monitored the networks performance and sent a transaction to update the fee factors when there was a change in the network load

+1

laynelafrance · 2021-12-02T03:06:27Z

flips/20211007-transaction-fees.md

+4. Transaction failed because the execution effort limit was reached.

-As discussed below, in scenario 1. the transaction shouldn't have been included in a collection in the first place. However since it was included and there is no way to charge the payer, the collection node account is charged with the transaction fees. The transaction fees are computed with the dynamic computation effort set to 0.
+As discussed below, in scenario 1. the transaction shouldn't have passed the access node in the first place. However since it has and there is no way to charge the payer, the access node account is charged with the transaction fees. The transaction fees are computed with the execution effort set to 0.


Flagging that we should discuss whether or not this is managed at AN or LN (collection node) level. I think currently we're trending toward AN but I do think we should complete the discussion on how failing txs without payer are managed between those 2 node types. I know @AlexHentschel has many thoughts ;)

joshuahannan

Great write-up! I mostly had typo fixes, but left a few other comments. Looking good so far! 😃

flips/20211007-transaction-fees.md

joshuahannan · 2021-12-06T17:28:31Z

flips/20211007-transaction-fees.md

+3. A potential third use of the effort cost factor would be to keep the cost of transaction stable according to the USD value of flow. This would require a periodic job to run, check the USD value of FLOW and update the cost factor to account for any changes. This would allow for the average transaction fees to stay relatively close to a set USD value.
+
+While the effort cost factor could be adjusted manually from the start, that would mean responding to network load would be very slow. A potential solution to adjusting the effort cost factor automatically would be a service that monitored the networks performance and sent a transaction to update the fee factors when there was a change in the network load.
+


I think this makes sense, but would this apply to inclusion effort and execution effort, or just execution?

flips/20211007-transaction-fees.md

…janez/transaction-fees

justinbarry · 2021-12-15T20:46:55Z

flips/20211007-transaction-fees.md

+
+The access node can check the balance of the payer prior to including their transaction and if the balance is below the minimum account balance, it would not include the transaction. However this prevents the payer from topping up his own balance, after an hones mistake. Some thought is needed here to address this edge case.
+
+The problem of providing economic incentive to include transactions of not part of this FLIP. This FLIP only provides a way to discourage access nodes to include transactions that cannot be paid for by the payer. 


Suggested change

The problem of providing economic incentive to include transactions of not part of this FLIP. This FLIP only provides a way to discourage access nodes to include transactions that cannot be paid for by the payer.

The problem of providing economic incentive to include transactions is not part of this FLIP. This FLIP only provides a way to discourage access nodes to include transactions that cannot be paid for by the payer.

laynelafrance

Awesome, awesome document @janezpodhostnik - thank you so much for proposing this 🙏

laynelafrance · 2021-12-16T18:49:33Z

flips/20211007-transaction-fees.md

+
+With an unaware payer (for example due to bad automation) or a malicious payer, that wants to drain the access node account or wants to cause excess traffic without paying transaction fees, it is in the access nodes best interest to not include these transactions.
+
+The access node can check the balance of the payer prior to including their transaction and if the balance is below the minimum account balance, it would not include the transaction. However this prevents the payer from topping up his own balance, after an hones mistake. Some thought is needed here to address this edge case.


hones -> honest

laynelafrance · 2021-12-16T19:01:41Z

flips/20211007-transaction-fees.md

+
+There are a few reasons why a transaction fails, which can be split into four categories, according to who finally gets charged for the transaction fees and how much they get charged.
+
+1. Transaction failed because the payers signature was incorrect, or the payer does not have enough funds to cover maximum transaction costs (transaction costs with the execution effort set to the execution effort limit).


payer has enough funds
-> payer doesn't have enough funds, I think you meant?

and +1

laynelafrance · 2021-12-16T19:46:44Z

flips/20211007-transaction-fees.md

+
+The calculation of inclusion effort can be done locally. 
+
+If we assume execution effort is 0 we get the minimum possible transaction fees. The maximum possible transaction fees can be computed with the execution effort set to the chosen execution effort limit.


For reference, can we share what the current computation limit is?

bluesign · 2021-12-20T21:10:17Z

Nice proposal, I know for sure you calculated, but what is the current situation with transaction fees vs staking rewards for nodes? How much percentage of the rewards coming from transaction fees?

I am curious how transaction fees will scale as network scales. Economic side of things are pretty interesting.

joshuahannan · 2021-12-20T23:07:21Z

@bluesign There is an event that indicates how many rewards come from fees every week and last week, that number was a bout 13.9 FLOW, so currently that puts it at about 0.001% coming from fees currently, but that number should go up as we enable this fee structure and the chain gets more and more usage.
reference:
https://flowscan.org/transaction/df669cd5b615708e54d5589761906ca5137d4860f1f78a11ea3bed48ff458e82?eventTypeFilter=A.8624b52f9ddcd04a.FlowIDTableStaking.EpochTotalRewardsPaid

redallica · 2021-12-21T11:17:13Z

How is the priority given to 2 different users aiming to get their transaction done ? Is it on a FCFS basis ? or other rules ? Hope it is not like Ethereum where it is basically designed as an auction where the one who pays more gets his transaction first.

j1010001 · 2021-12-21T23:46:26Z

How is the priority given to 2 different users aiming to get their transaction done ? Is it on a FCFS basis ? or other rules ? Hope it is not like Ethereum where it is basically designed as an auction where the one who pays more gets his transaction first.

Without going into details and edge cases on how network latency and other network attributes affect the transaction ordering, we can say that the transaction ordering is based on the time a transaction was received. Also, the fee is determined by the network based on the resources your transaction consumes, so a user can’t give their transactions priority by offering to pay more.

satyamakgec · 2022-01-11T11:06:37Z

flips/20211007-transaction-fees.md

+
+The ultimate goal from the protocols perspective is that transactions cost exactly the amount of FLOW needed to pay for the cost of the resources needed by the nodes to process the transaction. However measuring the exact resource consumption would be difficult and costly in itself, instead the goal is to make a reasonably good approximation and improve that approximation over time.
+
+From the users perspective the implementation of transaction fees should satisfy the following criteria:


What about the protocol safety perspective ? I would like to have those as the bullet points as well. That will give us a best path forward where we will not compromise the network security over the users easiness or vice-versa.

Protocol safety perspective -

Maximize the attack cost to prevent the network.

User would not be able to ddos so the computation/ effort cost always align with the transaction fee.

Surge cost to settle down the congestion within the network.

Do we want to support tipping or not ? As NFTs are tradable assets so getting in auction faster than others if it is time bounded auction then it will be a good usecase.

satyamakgec · 2022-01-11T11:09:35Z

flips/20211007-transaction-fees.md

+[^2]: Except when an explicit decision is made to change them.
+
+Transaction fees are deducted from the transaction payer's default Flow wallet automatically. This happens as the final step of transaction execution. If the transaction fails the fees are still deducted and no other state change (except the fee deduction) is committed.
+


Aren't we checking at the first step of transaction execution whether the account has sufficient balance to pay for the transaction fee or not ?

We are currently not checking this. The reason was that we wanted to allow a user with 0 funds to (during a single transaction) acquire funds (e.g. withdraw staking rewards) then use those to pay for the transaction fees.

In light of recent discussions the logic might change. If the users balance is below the minimum account balance (which should be significantly larger than the maximum theoretical transaction fees) do not execute the transaction at all.

satyamakgec · 2022-01-11T11:13:16Z

flips/20211007-transaction-fees.md

+
+
+The first separation of the transaction fees (<img src="https://render.githubusercontent.com/render/math?math=F">) is into inclusion fees (<img src="https://render.githubusercontent.com/render/math?math=F_i">) and execution fees (<img src="https://render.githubusercontent.com/render/math?math=F_e">). The inclusion fees roughly represent the cost of including the transaction in a collection and passing it from node to node. Inclusion fees must be designed in a way so they can be calculated without executing the transaction. The execution fees account for the approximate operational cost of executing the transaction script passing the results to the verification node and verifying it. Execution fees cannot be know without actually executing the transaction, but each transaction should specify a stopping limit for the execution fees.
+


IMO use computation time units instead of effort/work units.

satyamakgec · 2022-01-11T11:32:06Z

flips/20211007-transaction-fees.md

+
+The surge factor could be adjusted manually from the start, by manually monitoring the network and sending a transaction to update the surge factor. This would mean responding to network load would be very slow. 
+
+A first automated solution to adjusting the surge factor would be a service that would monitor the network load and send a transaction to update the surge factor when there was a change in the network load.


This is a chicken and egg problem where if the network congestion increases and to reduce this we are submitting the transaction to increase the surge factor so the load can be off burdened, but our transaction may not go through as well because of the "network congestion".

Even with the automated solution it has the same problem, I think the better way to make the network smarter where network keeps track of the block fullness target for the current block so the surge factor can change in the next block to consolidate the network activity appropriately. To avoid updating the surge factor too frequently we could use the time period to update the surge factor in a more deterministic manner.

The suggestion to use a service to keep track of load and update the surge factor is a bad one and I will update this.

The reason it's a bad one is that that service would be external to the network, so it could not be trusted.

Your suggestion to keep track of block fullness is a lot better. I think I will rewrite this so that surge control logic is executed during the system transaction/chunk (at the end of every block).

satyamakgec · 2022-01-11T11:34:17Z

flips/20211007-transaction-fees.md

+When automation will be in place for adjusting the surge factor, the surge factor could be adjusted frequently (in the span it takes to run a few blocks and detect a surge), but it should not change too drastically otherwise users cannot respond to the change.
+### Effort cost
+
+In general there would be two different effort cost functions, one for inclusion fees <img src="https://render.githubusercontent.com/render/math?math=F_I%20%3D%20p_I(I)"> and one for execution fees <img src="https://render.githubusercontent.com/render/math?math=F_E%20%3D%20p_E(E)">. This would allow for fine tuning what kind of transactions are the most cost optimal transactions. For example if the execution effort cost function is just a linear function, it makes sense to pack as much execution effort into one transaction as possible, but if the execution effort cost becomes quadratic at a certain effort threshold, it would make sense for users to break transactions into smaller transactions (if possible) that use effort up to that threshold.


Some how image URLs are not working ??

Seems to be working for me. The Known problem is that the equation/math images generated by: githubusercontent.com/render/math?math= are not visible on dark-mode as they have a transparent background. I have to look into fixing that.

satyamakgec · 2022-01-11T11:38:02Z

flips/20211007-transaction-fees.md

+
+In the first iteration of variable transaction fees the effort cost functions can simply be constant coefficients: <img src="https://render.githubusercontent.com/render/math?math=F_E%20%3D%20p_E%20*%20E"> and <img src="https://render.githubusercontent.com/render/math?math=F_I%20%3D%20p_I%20*%20I">. The coefficients are referred to as the inclusion effort cost parameter and the execution effort cost parameter, or together as effort cost parameters.
+
+The effort cost parameters should be defined on a smart contract and adjustable via the service account admin resource and should be accessible for everyone to read. 


My assumption is that service account will get changed to governance account in future as these operations are the destructive operations and one should not have the authority to change these values.

Yes the service account is the future governance account.

It is also possible that the control of these specific parameters might be split out to be controlled by a different account.

Thats up to future decisions.

Right now the service account is the one in control of transaction fee rates and storage fee rates (and other things, like epoch parameters) and is a multisig account, so I think this currently fits there.

…n-fees

2097: [FVM] Change fee deduction logic r=janezpodhostnik a=janezpodhostnik # Fee deduction logic change The [FLIP: Variable transaction fees](onflow/flow#660) specifies that transaction fees will be calculated as `Fees = surgeFactor ( inclusionEffortCost * InclusionEffort + executionEffortCost * executionEffort)`. This calculation happens in the FlowFees core contract with the following [PR](onflow/flow-core-contracts#279). The FVM needs to provide the inclusionEffort and the executionEffort. This PR: - imports and uses the changes to the the `FlowFees` core contract - transaction fee deduction was changed to use `FlowFees` - sets the fee parameters so the fees are still a static 0.0001 - sets the default (on bootstrap) `surgeFactor` to 1.0 - sets the default (on bootstrap) `inclusionEffortCost` to 0.0001 and the static `inclusionEffort` to 1 0000 0000 (or 1.0) - sets the default `executionEffortCost` to 0.0 and the `executionEffort` to "computation" (which will be renamed to executionEffort) The reason that the `executionEffortCost` is 0.0 and `executionEffort` is set to computation is that the computation usage will start appearing in the new `FlowFees.FeesDeducted` event emitted. When switching over to dynamic execution fees the `inclusionEffortCost` and `executionEffortCost` will change from 0.0001, 0.0 to 0.00005, something non-zero. ## Breaking change ⚠️ `TransactionFeeDeductionFailedError` No longer contains the amount of fees that were supposed to be deducted. That number is calculated within the `FlowFees.deductTransactionFee` contract function and is not available to the FVM. Co-authored-by: Janez Podhostnik <janez.podhostnik@gmail.com>

tx fees first version

ed54c82

janezpodhostnik self-assigned this Oct 7, 2021

minor fixes

c919cc5

vercel bot had a problem deploying to Preview October 8, 2021 16:15 Failure

second pass

6999d11

vercel bot deployed to Preview October 13, 2021 16:23 View deployment

updates

d176fdb

vercel bot deployed to Preview October 18, 2021 17:53 View deployment

diagram fix

ced5dc2

vercel bot deployed to Preview October 18, 2021 17:57 View deployment

fix image

97ff9d4

vercel bot deployed to Preview October 18, 2021 18:06 View deployment

update

4d02b36

janezpodhostnik marked this pull request as ready for review October 21, 2021 16:53

Merge branch 'master' of github.com:onflow/flow into janez/transactio…

93a3c54

…n-fees

vercel bot deployed to Preview November 17, 2021 16:02 View deployment

janezpodhostnik requested a review from laynelafrance November 18, 2021 17:40

redesign fee breakdown

2fc4364

vercel bot deployed to Preview November 23, 2021 19:05 View deployment

j1010001 approved these changes Nov 24, 2021

View reviewed changes

AlexHentschel reviewed Nov 26, 2021

View reviewed changes

laynelafrance reviewed Dec 2, 2021

View reviewed changes

update fees flip

0f09851

vercel bot deployed to Preview December 2, 2021 22:15 View deployment

wording changes

cdac8b5

vercel bot deployed to Preview December 3, 2021 19:56 View deployment

joshuahannan reviewed Dec 6, 2021

View reviewed changes

fixed spelling mistakes

d20f53e

j1010001 added the FLIP Flow Improvement Proposal label Dec 13, 2021

j1010001 requested a review from laynelafrance December 13, 2021 16:08

janezpodhostnik added 2 commits December 13, 2021 19:27

The diagram isn't needed

43cc9b2

Merge branch 'janez/transaction-fees' of github.com:onflow/flow into …

c804929

…janez/transaction-fees

vercel bot deployed to Preview December 13, 2021 19:28 View deployment

justinbarry reviewed Dec 15, 2021

View reviewed changes

laynelafrance approved these changes Dec 16, 2021

View reviewed changes

janezpodhostnik changed the title ~~Dynamic transaction fees~~ Variable transaction fees Dec 16, 2021

turbolent mentioned this pull request Dec 17, 2021

Determine expensive operations and adjust computation usage onflow/cadence#1221

Closed

satyamakgec reviewed Jan 11, 2022

View reviewed changes

Merge branch 'master' of github.com:onflow/flow into janez/transactio…

e364f55

…n-fees

vercel bot deployed to Preview January 11, 2022 22:54 View deployment

Wording changes

97eabf6

vercel bot had a problem deploying to Preview January 31, 2022 17:06 Failure

some wording changes

2f48991

vercel bot had a problem deploying to Preview February 1, 2022 22:01 Failure

pgebheim added the S-Governance label Feb 8, 2022

pgebheim mentioned this pull request Feb 16, 2022

Variable Transaction Fees: Execution Effort #753

Merged

joshuahannan changed the title ~~Variable transaction fees~~ FLIP: Variable transaction fees Feb 17, 2022

janezpodhostnik mentioned this pull request Mar 6, 2022

[FVM] Change fee deduction logic onflow/flow-go#2097

Merged

janezpodhostnik mentioned this pull request Mar 17, 2022

Introduce new transaction fee calculation onflow/flow-core-contracts#279

Merged

pgebheim merged commit c78658c into master Mar 23, 2022

This was referenced Oct 18, 2022

FLIP-660: Readability cleanups #1147

Closed

FLIP-660: Readability cleanups onflow/flips#37

Merged

	- If the protocol is experiencing a lot of traffic, all transactions should become more expensive, until the high traffic subsides.
	- Surge pricing: If the protocol is experiencing a lot of traffic, all transactions should become more expensive, until the high traffic subsides.


		3. A potential third use of the effort cost factor would be to keep the cost of transaction stable according to the USD value of flow. This would require a periodic job to run, check the USD value of FLOW and update the cost factor to account for any changes. This would allow for the average transaction fees to stay relatively close to a set USD value.

		While the effort cost factor could be adjusted manually from the start, that would mean responding to network load would be very slow. A potential solution to adjusting the effort cost factor automatically would be a service that monitored the networks performance and sent a transaction to update the fee factors when there was a change in the network load.

	While the effort cost factor could be adjusted manually from the start, that would mean responding to network load would be very slow. A potential solution to adjusting the effort cost factor automatically would be a service that monitored the networks performance and sent a transaction to update the fee factors when there was a change in the network load.
	While the effort cost could be adjusted manually from the start, that would mean responding to network load would be very slow. A potential solution to adjusting the effort cost factor automatically would be a service that monitored the networks performance and sent a transaction to update the fee factors when there was a change in the network load.

		For the first iteration the inclusion effort could be defined to be relative to the byte length of the (rlp encoded) transaction (scaled to match average execution effort) plus a well chosen constant part. The constant inclusion effort part could be chosen by observing the impact of transactions with a different byte size and a similar execution effort. This data could be used to predict how much impact a transaction that is 0 bytes long would make.

		Having a positive constant inclusion effort as well as a dynamic one means that splitting a transaction into smaller pieces does not make sense (considering only the inclusion part of the transaction fees). If this would be desired (from the protocols perspective) that the transactions are as small as possible or of a certain size, the dynamic inclusion effort could be defined to grow faster then linear with the byte size of the transaction.

		- <img src="https://render.githubusercontent.com/render/math?math=E_i"> Inclusion effort would be the byte size of a transaction plus a constant inclusion effort that would be picked by using linear regression on the impact of different size transactions (with the same execution effort) on the network, and extrapolating for a 0 size transaction.
		- <img src="https://render.githubusercontent.com/render/math?math=f_i">: Effort cost factor would be picked so that for 95% of transactions that have been seen so far the inclusion fees would cost less then `TC/2`.


		This separation into a part that can be know before the transaction is executed, and a part that cannot be know before the transaction is executed, allows for better oversight from the user into how much the transaction will actually cost.

		Each part of the fees can be further split into effort(<img src="https://render.githubusercontent.com/render/math?math=E">) and an effort cost factor(<img src="https://render.githubusercontent.com/render/math?math=f">) of FLOW per unit of effort. The effort required for a specific transaction should be independent of when the transaction is sent[^1] and should only depend on the properties of the transaction. On the other hand the effort cost factor would change over time depending on network activity.


		The access node can check the balance of the payer prior to including their transaction and if the balance is below the minimum account balance, it would not include the transaction. However this prevents the payer from topping up his own balance, after an hones mistake. Some thought is needed here to address this edge case.

		The problem of providing economic incentive to include transactions of not part of this FLIP. This FLIP only provides a way to discourage access nodes to include transactions that cannot be paid for by the payer.


		With an unaware payer (for example due to bad automation) or a malicious payer, that wants to drain the access node account or wants to cause excess traffic without paying transaction fees, it is in the access nodes best interest to not include these transactions.

		The access node can check the balance of the payer prior to including their transaction and if the balance is below the minimum account balance, it would not include the transaction. However this prevents the payer from topping up his own balance, after an hones mistake. Some thought is needed here to address this edge case.


		There are a few reasons why a transaction fails, which can be split into four categories, according to who finally gets charged for the transaction fees and how much they get charged.

		1. Transaction failed because the payers signature was incorrect, or the payer does not have enough funds to cover maximum transaction costs (transaction costs with the execution effort set to the execution effort limit).


		The calculation of inclusion effort can be done locally.

		If we assume execution effort is 0 we get the minimum possible transaction fees. The maximum possible transaction fees can be computed with the execution effort set to the chosen execution effort limit.


		The ultimate goal from the protocols perspective is that transactions cost exactly the amount of FLOW needed to pay for the cost of the resources needed by the nodes to process the transaction. However measuring the exact resource consumption would be difficult and costly in itself, instead the goal is to make a reasonably good approximation and improve that approximation over time.

		From the users perspective the implementation of transaction fees should satisfy the following criteria:

		[^2]: Except when an explicit decision is made to change them.

		Transaction fees are deducted from the transaction payer's default Flow wallet automatically. This happens as the final step of transaction execution. If the transaction fails the fees are still deducted and no other state change (except the fee deduction) is committed.



		The first separation of the transaction fees (<img src="https://render.githubusercontent.com/render/math?math=F">) is into inclusion fees (<img src="https://render.githubusercontent.com/render/math?math=F_i">) and execution fees (<img src="https://render.githubusercontent.com/render/math?math=F_e">). The inclusion fees roughly represent the cost of including the transaction in a collection and passing it from node to node. Inclusion fees must be designed in a way so they can be calculated without executing the transaction. The execution fees account for the approximate operational cost of executing the transaction script passing the results to the verification node and verifying it. Execution fees cannot be know without actually executing the transaction, but each transaction should specify a stopping limit for the execution fees.


		The surge factor could be adjusted manually from the start, by manually monitoring the network and sending a transaction to update the surge factor. This would mean responding to network load would be very slow.

		A first automated solution to adjusting the surge factor would be a service that would monitor the network load and send a transaction to update the surge factor when there was a change in the network load.


		In the first iteration of variable transaction fees the effort cost functions can simply be constant coefficients: <img src="https://render.githubusercontent.com/render/math?math=F_E%20%3D%20p_E%20%20E"> and <img src="https://render.githubusercontent.com/render/math?math=F_I%20%3D%20p_I%20%20I">. The coefficients are referred to as the inclusion effort cost parameter and the execution effort cost parameter, or together as effort cost parameters.

		The effort cost parameters should be defined on a smart contract and adjustable via the service account admin resource and should be accessible for everyone to read.

FLIP: Variable transaction fees #660

FLIP: Variable transaction fees #660

Uh oh!

Conversation

janezpodhostnik commented Oct 7, 2021

Description

Uh oh!

vercel bot commented Oct 7, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

janezpodhostnik commented Nov 23, 2021

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

AlexHentschel left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

joshuahannan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vercel bot commented Oct 7, 2021 •

edited

Loading