Accept Once Filter using Redis MetadataStore

References:

Raise below DO ticket(s) to configure Redis for your application:

https://jira-agile.mbopartners.com/browse/DO-9831
https://jira-agile.mbopartners.com/browse/DO-10211
https://jira-agile.mbopartners.com/browse/DO-10374

For Development

Raise below DO ticket(s) to get access to connect to Elasticache Redis

from local.

https://jira-agile.mbopartners.com/browse/DO-11243
https://jira-agile.mbopartners.com/browse/DO-10050

Verifying your access to redis (Only for development purpose)

After getting access to AWS, follow below steps to verify your access to
redis:

1. Login to AWS with your provided

credentials https://596212449348.signin.aws.amazon.com/console/
2. Enable MFA for your AWS account by
following https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credent
ials_mfa_enable_virtual.html
3. Navigate to https://us-east-1.console.aws.amazon.com/iam/home?
region=us-east-1#/identity_providers
4. Copy your "User" details which will be in
format arn:aws:iam::XXX:user/YYY
5. Copy and paste below script into a file in your local system
(example: aws_mfa_auth.sh)

#!/bin/bash

unset AWS_ACCESS_KEY_ID;unset AWS_SECRET_ACCESS_KEY;unset

AWS_SESSION_TOKEN;
unset tokens; unset x; unset mfa_device;
unset access_key; unset secret_key; unset session_token

mfa_device="arn:aws:iam::XXX:mfa/YYY"

echo "mfa_device set to ${mfa_device}"

echo "Enter One Time Passcode: "
read x
tokens=$(aws sts get-session-token --serial-number arn:aws:iam::XXX:mfa/YYY
--token-code ${x})

access_key=$( echo $tokens | jq -r '.Credentials| .AccessKeyId')

secret_key=$( echo $tokens | jq -r '.Credentials| .SecretAccessKey')
session_token=$( echo $tokens | jq -r '.Credentials| .SessionToken')

echo export AWS_ACCESS_KEY_ID=${access_key}

echo export AWS_SECRET_ACCESS_KEY=${secret_key}
echo export AWS_SESSION_TOKEN=${session_token}

echo -e "AWS_ACCESS_KEY_ID=${access_key}\nAWS_SECRET_ACCESS_KEY=$
{secret_key}\nAWS_SESSION_TOKEN=${session_token}" > ~/mfa.env

6. Update your User details in above script by replacing XXX and YYY
with proper values.
7. Execute above script in your terminal with command sh
aws_mfa_auth.sh
8. Enter your One Time Passcode from your configured identity provider
for AWS account.
9. You should be getting response similar to below output.

export AWS_ACCESS_KEY_ID=XXX
export AWS_SECRET_ACCESS_KEY=YYY
export AWS_SESSION_TOKEN=ZZZ

10. If you get the values then you are properly configured with
Redis. If not, please check with Dev-Ops team.

Connecting to redis using redis-cli

1. Download and configure redis-cli

2. RUN below java code with proper values to generate password

NOTE: import classses can be found

in https://github.com/mbopartnersinc/api-common-library/tree/main/api-
redis/src/main/java/com/mbopartners/api/common/redis/config

package com.mbopartners.api.common.redis.config.test;

import com.mbopartners.api.common.redis.config.IAMAuthTokenRequest;
import
com.mbopartners.api.common.redis.config.RedisIAMAuthCredentialsProvider;
public class RedisPasswordGenerator {
public static void main(String[] args) {
RedisIAMAuthCredentialsProvider
redisIAMAuthCredentialsProvider = new RedisIAMAuthCredentialsProvider();
IAMAuthTokenRequest iamAuthTokenRequest = new
IAMAuthTokenRequest();
iamAuthTokenRequest.setReplicationGroupId("preprod-
elasticache-redis");
iamAuthTokenRequest.setRegion("us-east-1");
iamAuthTokenRequest.setAccessTokenRefreshInSeconds("900");

redisIAMAuthCredentialsProvider.setAccessTokenRefreshInSeconds("900");
iamAuthTokenRequest.setUserId("XXX");
redisIAMAuthCredentialsProvider.setUserId("XXX");
redisIAMAuthCredentialsProvider.setSessionId("YYY");
redisIAMAuthCredentialsProvider.setAccessKeyId("ZZZ");
redisIAMAuthCredentialsProvider.setSecretAccessKey("WWW");

redisIAMAuthCredentialsProvider.setIamAuthTokenRequest(iamAuthTokenRequest)
;
redisIAMAuthCredentialsProvider.init();
System.out.println("PASSWORD: " +
redisIAMAuthCredentialsProvider.getIamAuthToken());
}
}

3. In below command, update XXX with username and YYY with

generated password with above java code and execute in terminal.

redis-cli -c -h clustercfg.preprod-elasticache-
redis.8stnx9.use1.cache.amazonaws.com -p 6379 --tls --user XXX -a 'YYY'

4. Test setting a key-value pair in redis using below command

SET key value

Output:

-> Redirected to slot [6918] located at preprod-elasticache-redis-0001-

002.preprod-elasticache-redis.8stnx9.use1.cache.amazonaws.com:6379
OK
(1.65s)
GET key
Output:

"value"
5.Now redis is ready for integration

Steps to integrate redis with non-vms-transformer

1. Add below dependencies to pom.xml

<dependency>
<groupId>com.mbopartners</groupId>
<artifactId>api-redis</artifactId>
 <version>0.0.55</version>
</dependency>
<dependency>
<groupId>org.springframework.integration</groupId>
<artifactId>spring-integration-redis</artifactId>
<version>5.5.0</version>
</dependency>
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>auth</artifactId>
<version>2.20.150</version>
</dependency>

After getting required access and creating required application based

redis configurations, follow below steps to integrate with api-redis with
your application.

2. Add below configurations to application.yaml

spring:

application:
name: non-vms-transformer
session:
store-type: ${SESSION_STORE_TYPE:none}
cache:
type: ${APP_CACHE_TYPE:simple}
redis:
key-prefix: ${REDIS_KEY_PREFIX:dev::}
use-key-prefix: true
redis:
host: ${REDIS_HOST:clustercfg.preprod-elasticache-
redis.8stnx9.use1.cache.amazonaws.com}
port: ${REDIS_PORT:6379}
ssl: true
cluster:
nodes: ${spring.redis.host}:${spring.redis.port}
aws:
userId: ${REDIS_USER_ID:xxxx}
accessKeyId: ${REDIS_CACHE_USERNAME:yyyyy}
secretAccessKey: ${REDIS_CACHE_PASSWORD:zzzzzzz}
replicationGroupId: ${REDIS_REPLICATION_GROUP_ID:preprod-elasticache-
redis}
region: ${REDIS_REGION:us-east-1}
sessionId:
accessTokenRefreshIntervalInSeconds: 900
connect-timeout: 100000
client-type: ${REDIS_CLIENT_TYPE:jedis}
filepoller:
metadatastore: redis
metadatakey: local::non-vms-transformer-input-file-metadata

NOTE: spring.redis.aws.sessionId needs to be populated for local development. DO NOT

CONFIGURE this field in any environment.

If we are using redis session, add below configurations to configure with redis
session.

spring:
 session:
store-type: ${SESSION_STORE_TYPE:redis}
redis:
flush-mode: on_save
cleanup-cron: "0 */30 * * * *"
configure-action: none
namespace: ${REDIS_KEY_PREFIX:dev::}${spring.application.name}
save-mode: always
3. Update spring.redis.aws.sessionId with AWS_SESSION_TOKEN
4. Update spring.redis.key-prefix based on env with format env::
5. Update spring.redis.aws.userId with data provided from Dev-Ops for each env
(refer https://jira-agile.mbopartners.com/browse/DO-10374)
6. Update spring.redis.aws.replicationGroupId with data provided from Dev-
Ops for each env (refer https://jira-agile.mbopartners.com/browse/DO-
10374)
7. If redis session is enabled, update 'spring.session.redis.namespace' with proper
env and app name.
8. Now Redis integration is completed.

Code changes in non-vms-transformer for implementing accept once filter

@Configuration
@Data
public class RedisMetadataStoreConfig {

@Value("${filepoller.metadatakey}")
private String filePollerMetaDataKey;

@Bean
public RedisMetadataStore redisMetadataStore(JedisConnectionFactory
jedisConnectionFactory) {
return new
RedisMetadataStore(jedisConnectionFactory,filePollerMetaDataKey);
}
}

@Bean
S3PersistentAcceptOnceFileListFilter acceptOnceFilter(){
return new
S3PersistentAcceptOnceFileListFilter(metadataStore,bucketName);
}

@Bean
public CompositeFileListFilter compositeFileListFilter(){
CompositeFileListFilter compositeFileListFilter = new
CompositeFileListFilter();
compositeFileListFilter.addFilter(acceptOnceFilter());
compositeFileListFilter.addFilter(new
S3RegexPatternFileListFilter(".*\\.csv$"));
return compositeFileListFilter;
}

@Bean
@InboundChannelAdapter(value = AppConstant.NONVMS_INBOUND_CHANNEL_ADAPTER,
poller = @Poller(fixedDelay = "${nonvms.job.fixed.delay.milliseconds}"))
public MessageSource<InputStream>
nonVMSTransformerInboundStreamingMessageSource() {
S3StreamingMessageSource messageSource = new
S3StreamingMessageSource(nonVMSTransformerTemplate());
messageSource.setRemoteDirectory(bucketName.concat(AppConstant.SLASH).conca
t(inFolderPath));
messageSource.setFilter(compositeFileListFilter());
messageSource.setMaxFetchSize(1);
return messageSource;
}

when running application, below logs can be seen

2024-07-24 23:24:50.845 INFO 58685 ---

[ main] .a.c.r.c.RedisIAMAuthCredentialsProvider : Using
credentials: XXX, preprod-elasticache-redis/?Action=connect&User=XXX&X-Amz-
Security-Token=YYY&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-
Date=20240724T175450Z&X-Amz-SignedHeaders=host&X-Amz-Expires=900&X-Amz-
Credential=XXX%2F20240724%2Fus-east-1%2Felasticache%2Faws4_request&X-Amz-
Signature=c7b52f685e0378da5dbda2acd86d7ad7217731d042b85b588d673f671c6783c4
2024-07-24 23:24:53.838 DEBUG 58685 --- [ main]
o.s.d.redis.core.RedisConnectionUtils : Fetching Redis Connection from
RedisConnectionFactory
2024-07-24 23:25:24.628 INFO 58685 --- [pool-1-thread-1]
com.mbopartners.api.logging.AppLogger : 2024-07-24T23:25:24.628908 - -
api-feature-flag-service -
com.mbopartners.api.common.redis.config.RedisConfig - - - Performing
Redis Re-auth

Disabling redis

To disable redis cache, update below properties

spring:
session:
# Disable redis session management
store-type: none
cache:
# Disable caching including redis. Update to SIMPLE to use in memory
cache
type: none
redis:
aws:
userId: ${REDIS_USER_ID:redis-app-dev-svc}
accessKeyId: ${REDIS_CACHE_USERNAME:xxx}
secretAccessKey: ${REDIS_CACHE_PASSWORD:yyy}
replicationGroupId: ${REDIS_REPLICATION_GROUP_ID:preprod-elasticache-
redis}
region: ${REDIS_REGION:us-east-1}
sessionId:
accessTokenRefreshIntervalInSeconds: 900
connect-timeout: 100000
# Update client-type to lettuce to stop redis config issue after
disabling redis
client-type: ${REDIS_CLIENT_TYPE:lettuce}

Enabling redis debug logs

Add below properties to enable redis debug logs

logging:
level:
org.springframework.data.redis: TRACE
io.lettuce: TRACE
redis.clients: TRACE
org.springframework.session: TRACE
org.springframework.cache: TRACE