Scribd
Upload
53 views

Assessment Developer

The document outlines a problem of allowing marketing agencies to securely upload data files to an AWS S3 bucket from different geographical locations for a data lake project. The assistant is tasked with designing an automated and scalable solution using Terraform that allows agencies to upload 1-24 CSV, Excel or JSON files between 20KB-50MB in size daily or weekly via SFTP. The solution must be deployed in the eu-west-1 region using AWS Free Tier services, follow security and infrastructure as code best practices, and programmatically alert if an agency misses a scheduled upload.

Uploaded by

Vijay Rajendiran
Copyright
© © All Rights Reserved
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
53 views

Assessment Developer

The document outlines a problem of allowing marketing agencies to securely upload data files to an AWS S3 bucket from different geographical locations for a data lake project. The assistant is tasked with designing an automated and scalable solution using Terraform that allows agencies to upload 1-24 CSV, Excel or JSON files between 20KB-50MB in size daily or weekly via SFTP. The solution must be deployed in the eu-west-1 region using AWS Free Tier services, follow security and infrastructure as code best practices, and programmatically alert if an agency misses a scheduled upload.

Uploaded by

Vijay Rajendiran
Copyright
© © All Rights Reserved
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
You are on page 1/ 1

INTERNAL

Problem:

As a global financial institution with operations we are spanning multiple AWS regions. As a Cloud
Engineer, you are responsible to help the organization by presenting, justifying, and implementing
simple solutions for simple problems and manageable/reproducible solutions for complex challenges.
We work with multiple marketing agencies and companies all around the world and now as part of data
lake project, we want to create a solution to those agencies to upload data to our S3 buckets. Being not
very tech savvy, they ask for SFTP connection for daily or weekly uploads.
Your task is to design a solution so those agencies in various countries can provide the data to our data
lake team.

Assumptions
1. File types are CSV, excel, and JSON. The smallest file we've seen so far is 20KB and the largest was
50MB.
2. Due to nature of data, each agency wants to upload 1-24 files in each batch.
3. Agencies are in different geographical locations and network latency for upload can be an issue.
4. Everyone in each agency will share the same SFTP user you create for that agency.

Requirements
1. Your solution should be deployed into eu-west-1 region. Use AWS Free Tier to Deploy your
solution.
2. Your code and documents should be hosted on a temporarily public GitHub repository
3. As a dynamic business line, the customer might want to start working with many new agencies
or say goodbye to an agency. Our SLA to onboard/offboard is 6 hours means your solutions must
be automated and reproducible.
4. All AWS resources (including IAM) or any type of resource configuration should be deployed and
managed using Terraform (no manual/custom solution through CLI or Console). The final
solution source code should be
easily maintainable and scalable.
5. If things go wrong we should be able to rollback and de-provision everything created during
setup. No broken window and cost incurring AWS resources should be left behind.
6. Principle of least privilege must be followed strictly and being explicitly documented / proven.
7. Incoming data contains PII so be careful of not exposing landing S3 bucket to the whole world!
8. Programmatically alert SRE team on missing data means if an agency was among daily providers
and didn't show up one day, this is considered a priority incident log and alert this incident on at
least one channel (email or Slack).
9. The deployment of new updates of over the solution, should be just a matter of updating the
repository.

Ask from You:


 Your solutions should follow "infrastructure as code" principles
 Security and DevSecOps mindset and clear documentation.
 Well-structured  git repository
 The end solution must be deployed and hosted on AWS. Choose any AWS service that you need.
 (extra points) frugality! Aim for keeping AWS bill as small as possible.

You might also like