Scenario:

You are a technical support engineer at Eightfold. Users have been reporting issues that they
are not able to apply to any job position from the company site (app.eightfold.ai/careers). This
issue has been affecting the user experience and causing frustration. Your task is to investigate
and resolve this problem.

Evaluation Rubric:

1.) Initial Analysis:

A.) As per the provided scenario statement, users are unable to apply to any position from the
company site. This is very generic problem statement and doesn’t provide us the clear picture.
There can be several underlying sub-scenario’s here. For example:

 Users is unable to sign-in/sign-up (for the first-time applicants) into the website.
OR
 As soon as users logs in successfully and clicks on the Job Designation a 5xx/4xx
error/Time-out error/Blank page shows up.
OR
 Users can login into the portal but Job Apply page doesn’t load or is terribly slow.
OR
 Users are able to get on to Job Apply pages however ‘Submit/Apply’ button
doesn’t work/ throws error.

So to find the exact issue, relevant questions must be asked.

B.) Potential Impact on the Applicant:

 For a candidate, applying of an opportunity is the very first step of the

recruitment process. In such a scenario, a non-functional website could really
tarnish the entire recruitment experience for the candidate.
 A non-functional website can dent the image of the organization for an
Applicant.
 It’s a loss of time and efforts for the applicant who was interested in the job
opportunity.

Business Impact:

 Loss of Talent, who could have been a great asset for the organization.
 Negative marketing of the organization.
  Clients would move on to other service providers, hence loss of business.

2. Troubleshooting:

 Take the Db backup and restore it into our Local Virtual Machine, so that
we will try to imitate the issue while being on the exact same version
number/patch number which customer is-on.
 Try to sign-up/login as Test user.
 If the website is not loading:

A.) Check the Developer Tool (F12) Console Or Network logs, to

check how the JavaScript is loading and which resource is taking how long to
load.

 As soon as users logs in successfully, and clicks on the Job Designation a

5xx/4xx error/Time-out error/Blank page shows up.

B.) 1.) Check the Developer Tool (F12) Network logs to check the
exact resource where error appears.
2.) In case of 5xx error, for example: 500 Internal Server Error,
502 Bad Gateway, 503 Service Unavailable, 504 Gateway timeout; reach out to
Hosting Team.
3.) In case of 4xx error, 400 Bad Request, 401 Unauthorized,
404 – Not Found, 403 Forbidden; reach out to Hosting Team.

 Users can login into the portal but Job Apply page doesn’t load or is terribly
slow.

C.) There can be various checks for this issue: Server Performance
check (for underpowered server), Server Configuration Check (for Caching issue), Slow DNS
lookup check, External Scripts Check (for plugin, widgets, trackers etc.),
Browser compatibility check.

 Users can get on to Job Apply pages however ‘Submit/Apply’ button

doesn’t work/ throws error.
D.) 1.) Do a inspect element on Elements tab in Developer tools
and try to find any syntax error or missing attributes.
2.) Check Network related issue which could be preventing the
submission such as Firewall restrictions.
3.) Check Event Handlers attached to submit button for any
conflicts.
4.) Clear browser cache and cookies.
 Relevant logs/Tools: HAR File, Splunk Tool, WebpageTest,
SolarWinds, GC Logs Analyzer, IIS.

3. Root Cause Identification:

After carefully examining all the logs and tools, we could see:
A.) 5xx Error Codes: Suggesting issues with Server configurations.
B.) Time-outs: High response times or time-outs indicates server -overloads.
C.) High- CPU usage: Warnings or errors related to memory, CPU usage, or disk
space points to resource exhaustion, leading to degraded performance or downtime.

4. Solution Proposal:

A.) Check server configurations (On Apache, IIS) to verify if it is correct or there
hasn’t been any inadvertent changes.
B.) Flush DNS Cache (on Windows: ipconfig /flushdns)
C.) Check Disk Space: Make Sure Enough disk space is available.
D.) Resource Availability Check: Add further resource Storage or upgrade to higher
instance type (in case of AWS): EC2: M4/M5a/ M5/M6a etc., if needed. ( Documentation has to
be referred and
relevant team (for example: Hosting team/Cloud Team) has to be Involved)
E.) Restart the server. (Windows Server Click Start Shut down on the Task bar
Choose Restart OK)

5. Customer Communication:

Draft an initial response acknowledging the user issue and ask questions to assign an
appropriate priority to the issue.

Dear <User’s Name>,

Good Afternoon and Greetings from Eightfold AI Support.

I’m Agam and We acknowledge the receiving of issue ‘Unable to Apply Job Vacancies
on (app.eightfold.ai/careers).’ We regret the inconvenience caused due to this incident and
apologies for the same.
We are currently looking into the concern however we would need following details
to investigate further:
A.) The issue appears for all the available job listings across all regions for all the
users? If it is specific to couple of users, please help us with the User Id’s.
 B.) Do you see any error? If yes, could you please provide error screenshot.
C.) When did you see the first error approximately.
D.) Were there any recent changes made? Any new Update/Code fix/
Enhancements/Version upgrades installed?

We will assign the case priority on the basis of details received from above question’s
answers.

Hope to hear from you soon.

Regards,
Agam
Eightfold AI.

-------------------------------------------------------------------------------------------------------------------------------
---------
Draft a concise and professional email response to the user, explaining your findings
and the proposed solution(s).

Dear <User’s Name>,

Greetings from Eightfold AI Support.

Thanks for the information provided in your last revert. It was very helpful for
conducting our investigation further.
Based on the information, we could deduce that the issue is related to Application
server. We could find some 5xx error in logs and hence we have taken following actions;

A.) Checked Server configuration for any mis-configs or errors.

B.) Flushed the DNS cache.
C.) We saw high average CPU usage in last 3 months, which indicates that we may
have to add some extra resources to keep the Application running smoothly and avoid down-
times. We are in touch
with our Hosting team to review this further and take appropriate actions on
above points.

Meanwhile, The server health is restoring quickly while, I’m writing this response
and soon it will attain its full health. Rest assured that we are taking all measure to avoid re-
occurrence of such scenarios in future.
We will keep you posted for further updates.

Regards,
Agam
 Eightfold AI.

-------------------------------------------------------------------------------------------------------------------------------
----------
Draft a message to respond to a user who identified one of the solutions you
proposed but raised a few concerns.

Dear <User’s Name>,

Greetings from Eightfold AI Support.

We understand your concern about the probable down-time of application you
indicated in your last revert, while upgrading the Application AWS resource; however please be
assured that it will require
minimal downtime (5-10 minutes) and that too will be carried out, of your business
hours. We are keeping any eye on server performance continuously.

Feel free to get back to us if you need any further assistance. We would be happy to
assist.

Regards,
Agam
Eightfold AI.

-------------------------------------------------------------------------------------------------------------------------------
-----------
Draft a message addressing the user’s concerns and provide an estimated timeline
for resolution.

Dear <User’s Name>,

Greetings from Eightfold AI Support.

This is to notify you about the concerns raised in your last response regarding some
of the users facing slow loading of website.
We got in touch with such users and found that there were two main reasons of
such slowness A.) Outdated Browser version B.) plugins/third party scripts added into there
browsers, which was interfering.
We requested them to update the browser version and disable two identified
plugins. Things have been back to normal.

Another update would be: As notified earlier, we have planned to elevate the AWS
EC2 instance storage and its type to ensure higher speed, reliability and
computation power. We have already raised an internal case on your behalf to our
hosting team. Case number is (SFxxxxxx9087), As per the latest updates from the team it may
take 48 business hours to
approve and action the changes proposed.

Regards,
Agam
Eightfold AI

-------------------------------------------------------------------------------------------------------------------------------
---------------
Draft a message to address user concern on progress delay as well. User is frustrated
at this time and is escalating to your Exec.

Dear <User’s Name>,

Greetings from Eightfold AI Support.

I understand your frustration and apologize for the delay in resolving your issue.
Please accept my sincere apologies for any inconvenience this has caused you. We
are actively working on resolving your concern, and I want to assure you that it is our top
priority. We are only left with
upgrading the Application resources for you, which is taking time more than
anticipated. Our technical team is currently investigating it for any relevant checks in the
hosting environment and want to ensure that everything is on place.
I appreciate your patience and understanding during this process. I will personally
keep you updated on the progress and provide you with any new information as soon as it
becomes available.
If there's anything else you'd like to share or if you have any further questions,
please don't hesitate to reach out to me directly.

Regards,
Agam
Eightfold AI

6. Preventive Measures to Development or Operations Team

 Please keep an eye on Average resource utilization in our Cloud Dashboard to
foresee such issue. You can also set alerts for critical thresholds.
 Involve such Customer environments in your weekly team meet, so that such
alerts don’t go un-noticed.

 Conduct Regular Load-testing.

 Utilize load balancers (if not already)

 Implement automated backup solutions for data and configurations to ensure

quick recovery.