We need to add data retention support to Core/Enterprise to provide similar functionality that was provided in previous implementations of InfluxDB. This is functionality to automatically delete data after it has hit a certain age.

Note: I am calling for days and not hours below - but this needs validated. We did have hours in previous versions. Trying to gather some data to validate this need. Deleting data with compaction also running on Enterprise has the opportunity for conflict - so I want to see if this can be minimized by limiting deletion of expired data to once per day.

MVP General Requirements
(Database Level Retention)

• At a database level, a customer must be able to set a retention period (number of days)
• The policy must have options for a number of days (delete data older than) or "never" - to allow unlimited retention
• The user must be able to change a retention policy that they previously set; however, it is not expected that any data would be recovered if they changed, for example, from 7 days to 21 days. Any data over 7 days (in this example) that has already been deleted does not have to be recoverable.
• When data is older than than the retention policy, the data should begin to be deleted; however it is not expected that it will be an immediate deletion and may be deleted in batch
• Determining the data age must be based on the timestamp of each record, and not "when" the data arrived; meaning - if the retention days is set at 30, and a late arriving record comes in with a timestamp that is already 29 days old, that record, despite being just added, will be eligible for deletion
• When a user is configuring a retention period and is setting it lower than its current setting, the user should be warned that it could result in existing data being deleted, and they should confirm this is what they want to do.

Post-MVP General Requirements
(Table Level Retention)

• At a table level, a customer must be able to set a retention period
• The policy must have options for a number of days (delete data older than) or "never" - to allow unlimited retention
• The policy must not supercede the max policy of the database; meaning - if the database is set to 90 days, then the table retention cannot be set to higher than 90 days (but it can be set lower, or can be set to any value if the database is set at "never").
• The user must be able to change a retention policy that they previously set; however, it is not expected that any data would be recovered if they changed, for example, from 7 days to 21 days. Any data over 7 days (in this example) that has already been deleted does not have to be recoverable.
• When data is older than than the retention policy, the data should begin to be deleted; however it is not expected that it will be an immediate deletion and may be deleted in batch
• Determining the data age must be based on the timestamp of each record, and not "when" the data arrived; meaning - if the retention days is set at 30, and a late arriving record comes in with a timestamp that is already 29 days old, that record, despite being just added, will be eligible for deletion
• When a user is configuring a retention period and is setting it lower than its current setting, the user should be warned that it could result in existing data being deleted, and they should confirm this is what they want to do

(other non-MVP Requirements)

• Users must be able to set the database retention as part of the create database function (as additional parameters), vs. a separate step
• Users have an API that allows them to set the retention periods (for MVP, CLI is satisfactory)

Global RP: Core & Enterprise
Table RP: Enterprise