Overview
The import and export feature uses the native RDB snapshot feature of Redis to import data into or export data out of a Memorystore for Redis instance. The use of the native RDB format prevents lock-in and makes it very easy to move data within Google Cloud or outside of Google Cloud. Import and export uses Cloud Storage buckets to store RDB files.
The behavior of a Memorystore for Redis instance during export is very similar to the BGSAVE command in open source Redis. Memorystore for Redis imports by copying your RDB file into your instance's data directory.
Import feature behavior
For optimal performance, you should import from Cloud Storage buckets located in the same region as your Redis instance.
Your Redis instance is unavailable during an import operation. Your instance becomes available after the import operation finishes.
If an import is successful, the existing data in the instance is overwritten by the data in the RDB file.
If an import fails for any reason, the instance is brought online, but the data may be fully flushed. You can retry the import using the same RDB file, or you may use another RDB file to restore data.
Your Memorystore for Redis instance can import RDB files from the same Redis version and from a previous Redis version, but it cannot import RDB files from a newer version.
An import operation can be cancelled, but you should note that the cancellation results in your instance recovering with a fully flushed cache. This is the case for both Basic Tier and Standard Tier instances.
An instance can only import backups from older Redis versions. An instance running Redis 5.0 can import an RDB exported from Redis 4.0, but an instance running Redis 4.0 cannot import an RDB from Redis 5.0.
Export feature behavior
You can read and write data to your Redis instance during an export operation; however, you cannot perform any admin operations like scaling, updating, or configuring your instance.
Your instance may experience increased latency during the export.
For optimal performance, you should export your RDB backups to Cloud Storage buckets located in the same region as your Redis instance.
During an export, the RDB file is saved to a Cloud Storage bucket of your choosing.
You can cancel an export operation at any time. Cancelling does not impact the data or availability of an instance.
Cancelling an export deletes the RDB file being written to the Cloud Storage bucket, and frees up memory used by the
BGSAVEprocess.
An export can overwrite an existing file with the same name with no extra permissions when using the Google Cloud console. To export over an existing file with
gcloud, the user account needs thestorage.objects.deletepermission.
Exporting under high write load
If you run an export during a period of high write-load your instance memory usage can increase up to 2X, causing the export operation to fail. You should perform export operations during periods of low writes. To monitor how much Redis data is stored in your instance you can create a dashboard to monitor your memory usage and monitor the instance memory usage metrics with Google Cloud's operations suite.
Memorystore for Redis export uses the BGSAVE feature of Redis to take a snapshot of the data in an instance. When executing BGSAVE, Redis forks a new child process to take the snapshot. Redis utilizes copy-on-write during this process.
This means that the initial fork allocates minimal extra memory, but any pages changed by Redis server write traffic are duplicated. Therefore, under periods of high writes, in the worst case, the memory footprint of Redis doubles, causing the export operation to fail.
Exporting during a BGSAVE operation
You cannot export while a BGSAVE operation is running. See for more details
Required permissions for importing and exporting
A user needs to configure sufficient IAM permissions to import or export RDB files. The simplest permissions setup is to apply the Cloud Memorystore Redis Admin and Storage Admin IAM roles to the user account, however this setup may be inappropriate for some user accounts because it gives them permissions to create, edit, or delete resources they shouldn't have access to. For instructions on setting minimally restricted permissions see Granting restricted permissions for import and export.
Duration of imports and exports
The duration of the import or export depends on the instance size and the size of the data set in the instance. Depending on the size, the imports and exports can take anywhere from a few minutes to a few hours.
For example, if you import or export a 10 GB instance with 8 GB of used memory, the operation can take 5 minutes. If you import or export a 120 GB instance with 100 GB of used memory, the operation can take an hour.
Monitoring progress of the operation
The time estimate for completion is not available while doing import and export. You can check whether the operation is in progress by checking the instance details page. To view the instance details page, click your instance ID on the Memorystore for Redis instances list page.
You can also monitor the progress of an import operation by running the following command:
gcloud redis instances describe [INSTANCE_ID] --region=[REGION]
Where:
[INSTANCE_ID]is the ID for your Memorystore for Redis instance.[REGION]is the region your instance is in.
The command outputs state: IMPORTING or state: READY as one of the description fields.
Unsupported Cloud Storage buckets
Requester Pays Cloud Storage buckets are not supported for Memorystore imports and exports.
Securing your Cloud Storage buckets
You should ensure that your Cloud Storage bucket has the right restrictions to prevent unauthorized access to the Redis RDB backup it contains. Please refer to Cloud Storage Access Control Options on how to secure your Cloud Storage buckets.
Ensuring sufficient memory to complete an export
To prevent export from causing memory starvation, you should have 50% free memory in the instance during the export process. This may not be required all the time, depending on your write load, but having 50% free memory ensures that there is enough free space to accommodate copy-on-write overhead, if many pages are touched during the export process.
To understand the current memory usage of your instance, set up a Google Cloud's operations suite alert to notify you when your memory usage reaches a customizable threshold. Additionally you should set up a memory monitoring dashboard with Google Cloud's operations suite. These measures allow you to understand your instances memory usage patterns. Understanding memory usage patterns enables you to size your instance correctly when exporting at regular intervals.
Exporting and importing to instances with read replicas
For all Redis instances, including read replica enabled instances, data is exported from the primary node.
For read replica enabled instances, exporting can increase the CPU load and memory footprint on the primary node.
When importing data into a read replica enabled instance, the primary restarts during the import operation, and the instance is unavailable until the new data is fully loaded into the database.
Export data from a Redis instance
Exporting your Redis instance creates an RDB backup file of your data. The export operation stores the RDB backup in a Cloud Storage bucket. Your instance's performance may be temporarily slower for the duration of the export.
For instructions on how to schedule RDB exports using Cloud Scheduler, see Scheduling Memorystore for Redis database exports using Cloud Scheduler.
Before you begin
Have a Cloud Storage bucket. We recommend that you export to a bucket located in the same region as your Redis instance.
To create a Cloud Storage bucket, see Creating storage buckets.
Have sufficient permissions to perform the export.
Export an RDB backup file to a Cloud Storage bucket
See INSRUCTIONS (for console & gcloud CLI).
Stop an ongoing export operation
See INSTRUCTIONS (for console & gcloud CLI).
Instance availability and behavior while exporting
During the export process you can read and write to your Redis instance, however you cannot run admin operations on your instance such as scaling or upgrading.
Your instance may experience increased latency during the export operation.
The export can take anywhere from a few minutes to a few hours. For example, if you export a 10 GB instance with 8 GB of used memory, the export can take 5 minutes, and if you export a 120 GB instance with 100 GB of used memory, the export can take an hour. To monitor the progress of the export, check the instance details page, which displays the status of the operation.
Import data from a Redis instance
Importing a Redis backup allows you to migrate data from another instance or restore your instance data from a previous RDB backup.
Before you begin
Have a Cloud Storage bucket that contains the RDB backup file that you want to import. We recommend that you import from a bucket in the same region as your Memorystore instance.
To create a Cloud Storage bucket, see Creating storage buckets.
To upload a file into a Cloud Storage bucket, see Export data from a Redis instance topic or Uploading Objects.
Have sufficient permissions to perform the import.
Import an RDB backup file into a Redis instance
See INSTRUCTIONS (for console & gcloud CLI).
Stop an ongoing import operation
See INSTRUCTIONS (for console & gcloud CLI).
Instance availability and behavior while importing
Importing overwrites all current instance data, so make sure that you are okay with your data being overwritten before you start an import.
Your Redis instance is unavailable during the import process. You cannot access instance data or modify the instance until the import is complete.
The import can take anywhere from a few minutes to a few hours. For example, if you import an 8 GB RDB file into a 10GB instance, the import can take 5 minutes, and if you import a 100GB RDB file into a 120GB instance, the import can take an hour. To monitor the progress of the import, check the instance details page, which displays the status of the operation.
An instance can only import backups from older Redis versions. An instance running Redis 5.0 can import an RDB exported from Redis 4.0, but an instance running Redis 4.0 cannot import an RDB from Redis 5.0.
Scheduling RDB exports using Cloud Scheduler
It’s possible to use Cloud Scheduler and Cloud Functions to automatically export a Memorystore for Redis database to Cloud Storage.
This configuration includes the following Google Cloud components:
Cloud Scheduler jobs: Jobs to make calls on a set schedule to start the database export.
Cloud Functions: Functions to export the data from Memorystore to Cloud Storage.
Pub/Sub messages: Messages sent and received for each data export event.
Cloud Storage buckets: Bucket to store the exported data.
Memorystore for Redis: Source database to export the data from.
A Cloud Scheduler jobs posts a message on a Pub/Sub topic with information about the Memorystore instance ID, the project ID, the region where it's located, and the Cloud Storage location at which to store the backup. This event triggers a Cloud Function that gets this payload and starts a database export on Memorystore for Redis through its API. The database generates the export and saves it to Cloud Storage. The following diagram shows this workflow.
Costs
This tutorial uses the following billable components of Google Cloud:
To generate a cost estimate based on your projected usage, use the pricing calculator.
Configure Scheduling
In the Google Cloud console, on the project selector page, select or create a Google Cloud project;
Make sure that billing is enabled for your Cloud project;
Activate Cloud Shell in the Google Cloud console;
Enable the Memorystore for Redis, Cloud Functions, Cloud Scheduler, and App Engine APIs;
Create a Cloud Storage bucket (terraform google_storage_bucket) and grant your Memorystore service account the permissions to export data to Cloud Storage with the Simple Storage Role;
Create a Pub/Sub topic, a Cloud Function, and a Cloud Scheduler job;
Create a Cloud Scheduler job (terraform google_cloud_scheduler_job).
