(Migration from Singapore to Mumbai)
Problem Statement
In Indus, we recently migrated from AWS Singapore region to Mumbai region as all our customer base is from India. The only service which is left in Singapore region that needs to be migrated to Mumbai region is AWS Mobile Analytics (AMA). The reason we can’t directly migrate it to Mumbai is because of the limitation at AMA side. (Once you create AMA endpoint to S3 bucket in the region it remains with it and can’t be changed). We have the data pipeline which reads this S3 bucket and puts the data into Redshift which is again in Singapore. But due to this, we are incurring huge data transfer cost (approx. $7K).
To avoid this we worked with AWS SA and found that since we can’t change the endpoint for AMA, our only option is to change the app on all those million devices in the market which will send the data to different AMA endpoint but then again we will be in the same situation in case we need to change this Endpoint in future when our handsets get distributed globally.
Another approach for us is to use “Kinesis Firehose” but again for the time being, it is not available in Mumbai Region.
Solution
We are planning to use Kinesis which will be the new endpoint for all our devices. Consumer Apps running on EC2 will then pull data from Kinesis Shards and put it into S3 bucket. This setup will be configured in Mumbai region. When it is ready, we will send an update to all our devices. Till the time all devices are upgraded, this setup will run in parallel with Singapore region AMA. AMA will be eventually shut down once all devices are upgraded.
Migration Plan of Action
Kinesis
In Mumbai region, we will set up new Kinesis with few shards (These shards will be configured to auto scale as we send the upgrade to all devices). Shard will then work with the consumer who is configured on EC2 instance and upload the data to the respective S3 bucket. We will use the Kinesis library to make sure that data from each shard is uploaded to the respective S3 bucket.
S3 Replication
Since older than two weeks data is not needed, we will set up the S3 cross region replication (Singapore to Mumbai) and allow it to run for at least month. This will populate needed data in Mumbai region before we start migrating data pipeline and Redshift.
Redshift
There is huge data (6TB) which resides in Redshift which currently runs in Singapore. We will be enabling cross-region snapshot backup. With this, we will have the required data in Mumbai region for data pipeline to work.
For more, visit www.techpartner.in or contact us at info@techpartner.in.
