Data Retention and Persistent Caps

Overview

Core Data service

In use cases, since core data persists data in the local database indefinitely, there is a need to persist the latest recent readings only and clean up the old ones, as keeping an infinite number of readings is considered computationally expensive and can lead to a lack of machine memory. Thus, a retention mechanism is placed on core data to keep a certain number of readings.

Under this mechanism, the maximum readings capacity is called MaxCap and the minimum readings capacity is called MinCap. Core data will create an internal schedule according to the Interval configuration to check if the number of readings are higher than the MaxCap. When the number of readings reach the MaxCap, Core data will purge the amount of the readings to the MinCap.

For example, the MaxCap is set to 10, the MinCap is set to 2 and the Interval is set to 3s. Now, core data will check how many readings are in the local database every 3 seconds. When the number of readings reach 10, core data will check the 3rd reading to find the related event's origin timestamp and perform function DeleteEventsByAge to delete events by age. This way the related readings will also be deleted.

Support Notifications service

As with core data, the number of notification can increase indefinitely, so the same retention mechanism can be also used to keep the latest notification and clean up the old ones.

Introduction

For detailed information on the data retention see Use Case for capping readings in Core Data.

Prerequisite Knowledge

• For detailed information on the data retention see Core Data Configuration Properties and browse to retention tab.

• For detailed information on the data retention see Notifications Configuration Properties and browse to retention tab.

Enable Data Retention

Two ways to enable data retention mechanism:

• Using environment variables to override the default configuration

  RETENTION_ENABLED: true 
  RETENTION_INTERVAL: <interval>  
  RETENTION_MAXCAP: <maxcap>
  RETENTION_MINCAP: <mincap>  
  

For detailed information about environment variables override see Service Configuration Overrides.

• Using Consul to override the default configuration Open Consul web user interface and go to Retention section of the core-data or support-notification service to customize the configuarion.

For detailed information about Consul see Configuration and Registry Provider.