DAV

1. List and explain different key roles for successful data analytics.

Ans: -

1. Data Analyst

• Role: Interprets data, analyses results, and provides actionable insights.

• Tasks: Perform data cleaning, run statistical analyses, create reports, and dashboards.

• Skills: SQL, Excel, Python/R, visualization tools (Tableau, Power BI).

2. Data Scientist

• Role: Builds models and algorithms to predict outcomes and uncover patterns.

• Tasks: Develop machine learning models, perform deep statistical analysis, experimental design.

• Skills: Machine learning, deep learning, advanced statistics, Python/R, big data tools.

3. Data Engineer

• Role: Designs, builds, and manages the data infrastructure.

• Tasks: Build data pipelines, ensure data quality, integrate data from multiple sources.

• Skills: SQL, ETL tools, Hadoop, Spark, cloud platforms (AWS, Azure, GCP).

4. Business Analyst

• Role: Acts as a bridge between business needs and technical teams.

• Tasks: Gather business requirements, translate them into data tasks, and interpret analytical
results in a business context.

• Skills: Communication, problem-solving, SQL, Excel, domain knowledge.

5. Data Architect

• Role: Designs the overall data structure and ensures it supports business needs.

• Tasks: Create database designs, oversee data storage and retrieval, ensure data security and
scalability.

• Skills: Database systems (SQL, NoSQL), system design, cloud architecture.

6. Machine Learning Engineer

• Role: Focuses on putting machine learning models into production.

• Tasks: Optimize and scale models, deploy models, monitor model performance.

• Skills: Python, TensorFlow/PyTorch, model deployment tools (Docker, Kubernetes).

7. Chief Data Officer (CDO) / Data Manager

• Role: Leads data strategy at the organizational level.

• Tasks: Define data governance policies, align data initiatives with business goals, ensure regulatory
compliance.

• Skills: Leadership, data governance, regulatory knowledge (GDPR, HIPAA), strategic planning.

8. Data Visualization Specialist

• Role: Converts complex results into clear, understandable visuals.

• Tasks: Build dashboards, create infographics, enhance storytelling through data.

• Skills: Visualization tools (Power BI, Tableau, D3.js), design thinking, user experience.

9. Domain Expert / Subject Matter Expert (SME)

• Role: Provides in-depth knowledge of the specific business area.

• Tasks: Ensure the relevance and applicability of data analytics solutions.

• Skills: Deep domain expertise, collaboration, critical thinking.

10. Data Governance Specialist

• Role: Ensures proper management, usage, and protection of data.

• Tasks: Manage data policies, ensure ethical use of data, maintain data integrity and privacy.

• Skills: Data privacy laws, compliance, auditing, metadata management.

Summary:
Successful data analytics is not just about technical skills — it requires a team effort blending technology,
business understanding, and clear communication.
Q.2. what is stepwise regression? explain its type

Ans: - Alright! Let’s go step-by-step:

What is Stepwise Regression?

Stepwise Regression is a method of building a regression model by automatically selecting which independent
variables (predictors) should be included.
Instead of manually trying combinations, the algorithm adds or removes predictors based on certain criteria like:

• p-values (statistical significance),

• AIC (Akaike Information Criterion),

• BIC (Bayesian Information Criterion), or

• Adjusted R².

Goal:
Find a model that is both accurate and simple (no unnecessary variables).

Types of Stepwise Regression:

There are three main types:

1. Forward Selection

• How it works:

1. Start with no variables in the model.

2. Add the most significant variable (the one with the lowest p-value).

3. Add variables one by one, each time picking the next most significant one.

4. Stop when no more variables meet the inclusion criteria (e.g., p-value < 0.05).

• Used when:
You have many predictors and want to build up the model gradually.

2. Backward Elimination

• How it works:

1. Start with all possible variables included.

2. Remove the least significant variable (highest p-value).

3. Keep removing variables until all remaining ones are statistically significant.

• Used when:
You suspect most variables matter, but you want to prune unnecessary ones.
3. Bidirectional Elimination (Stepwise Selection)

• How it works:

1. Combines Forward and Backward steps.

2. Start either with no variables (like forward) or all variables (like backward).

3. After adding a new variable, check if any previously included variable has become non-
significant (and remove it).

4. Continue adding and removing variables until no more changes improve the model.

• Used when:
You want a more flexible approach — allowing additions and removals at each step.

Bonus:

• Advantages:
Automates model selection, saves time.
Reduces model complexity (removes irrelevant variables).

• Disadvantages:
Can overfit if not careful.
May miss the best model because it's based on greedy choices at each step (not looking at all
combinations).

Q.3. What is TF-IDF ?

Ans: - Let's break it down clearly:

TF-IDF stands for Term Frequency–Inverse Document Frequency.

It is a numerical statistic used to measure how important a word is in a document relative to a collection of
documents (corpus).

It is very popular in:

• Information Retrieval (like search engines)

• Text Mining

• Natural Language Processing (NLP)

Why TF-IDF?

• Some words (like "the", "is", "and") appear very often but don't tell us much about the topic.
 • TF-IDF down-weights common words and up-weights rare but important words.

In short:
Frequent in one document + rare across others = Important.

TF-IDF Formula:

TF-IDF is a combination of two things:

1. Term Frequency (TF)

• How often a word appears in a document.

2. Inverse Document Frequency (IDF)

• How unique or rare the word is across all documents.

Simple Example:

• Word "learning" appears in all documents → Low IDF (common word).

 • Word "machine" appears only in Doc1 → High IDF (rare, important for Doc1).

Thus:

• "machine" will have a high TF-IDF in Doc1.

• "learning" will have a lower TF-IDF because it's common everywhere.

Q.4. Difference between matplotlib and seaborn library

Ans: -

Q.5. Explain the components of time series

Ans: - Great! Time series data is everywhere — stock prices, weather data, website traffic, etc. To analyse it
effectively, we break it down into four main components:

Components of Time Series

1. Trend (T)

• Definition: The long-term direction in the data (upward, downward, or flat).

• Example: A company’s revenue steadily increasing over several years.

• Looks like: A smooth curve or line showing general growth or decline.

• Why it matters: Helps understand overall growth or decline beyond short-term fluctuations.

2. Seasonality (S)

• Definition: Regular, repeating patterns or cycles at fixed time intervals (hourly, daily, weekly,
monthly, yearly).

• Example: Ice cream sales spike every summer.

• Looks like: A repeating wave or periodic pattern.

• Why it matters: Shows predictable patterns due to calendar effects (holidays, seasons).

3. Cyclic Component (C)

• Definition: Fluctuations that occur over longer periods but not at fixed intervals.

• Example: Economic booms and recessions.

• Difference from seasonality: Not tied to a calendar — more irregular and long-term.

• Why it matters: Helps spot broader economic or structural patterns.

4. Irregular or Residual Component (I)

• Definition: The random noise or unpredictable variation left over after removing trend,
seasonality, and cycles.

• Example: Sudden sales drop due to a one-time event like a natural disaster.

• Why it matters: Not forecastable; helps estimate uncertainty and risk.

Q.6. Explain Arima in details also state its pros and cons

Ans. Absolutely! Let’s dive deep into ARIMA, one of the most popular models in time series forecasting.

What is ARIMA?

ARIMA stands for:

AutoRegressive Integrated Moving Average

It’s a powerful statistical model used to analyse and forecast univariate time series data — data with a single
variable observed over time.

ARIMA Components Explained:

1. AR (AutoRegressive) [p]

• Predicts the future based on past values.

• Example: "Today’s value depends on the last few days’ values."

• Order p = number of lag observations included.

2. I (Integrated) [d]

• Refers to the differencing of data to make it stationary (i.e., remove trend/seasonality).

• Differencing = subtracting the previous value from the current one.

• Order d = number of times the data is differenced.

3. MA (Moving Average) [q]

• Predicts the future based on past errors (residuals).

• Example: "Today’s error is similar to a combination of past errors."

• Order q = number of past forecast errors in the prediction.

ARIMA Notation:

ARIMA(p,d,q)ARIMA(p, d, q)

• p = number of AR terms (lags of Y)

• d = number of differencing steps

• q = number of MA terms (lags of residuals)

How ARIMA Works (Step-by-Step):

1. Check for stationarity (mean and variance don’t change over time).

2. Make the series stationary using differencing (based on d).

3. Identify p and q using ACF (Autocorrelation Function) and PACF (Partial ACF) plots.

4. Fit the ARIMA model with chosen (p,d,q).

5. Validate model performance using metrics (RMSE, AIC).

6. Forecast future values.

 Example:

Let's say:

• You have monthly sales data.

• You notice a trend but no seasonality.

• After differencing once, the data becomes stationary.

• ACF and PACF plots suggest 1 AR term and 1 MA term.

Your model might be:

ARIMA(1,1,1)A

Advantages (Pros) of ARIMA:

Advantage Description

Handles TrendWorks well with data that shows a consistent upward or downward trend.

Strong for short-term forecasting with a single variable.

Forecasting Power

Flexible Can model many real-world time series with just (p,d,q).

Easy to analyze residuals, check model fit using ACF/PACF, AIC/BIC, etc.
Clear Diagnostics

Disadvantages (Cons) of ARIMA:

Disadvantage Description

Univariate Only Can't directly handle multiple variables (unless extended to ARIMAX).

Needs the data to be stationary — requires pre-processing.

Stationarity Required

Struggles with strong seasonal patterns (use SARIMA instead).

Poor for Seasonality

Assumes linear relationships — not great for complex, nonlinear patterns.

Linear Assumptions

Outliers can distort predictions significantly.

Sensitive to Outliers

Variants of ARIMA:

• SARIMA: Seasonal ARIMA → handles seasonality.

• ARIMAX: ARIMA with exogenous variables (extra inputs).

• SARIMAX: Seasonal ARIMA with exogenous variables.

 When to Use ARIMA:

• You have a single time series.

• There's a clear trend, but seasonality is weak or removed.

• You want a statistical, interpretable model (not black-box like some ML methods).

Would you like a code example (in Python using statsmodels) to see how ARIMA is implemented in practice?

Q.7 . Explain in detail seven practice areas of text analytics.

Ans: - Certainly! Text analytics involves extracting meaningful information from unstructured text data. It’s
widely used in business, healthcare, social media, and more. Below are the seven key practice areas of text
analytics explained in detail:

[A] Seven Practice Areas of Text Analytics

1. Information Extraction (IE)

• Goal: Automatically extract structured data from unstructured text.

• Tasks Include:

o Named Entity Recognition (NER): Identifying names of people, organizations, dates,

locations.

o Relationship Extraction: Finding how entities are related.

o Event Extraction: Detecting events and actions (e.g., "Company A acquired Company B").

Use Case: Extracting company names and acquisition dates from news articles.

2. Document Classification (Text Categorization)

• Goal: Assign predefined labels or categories to text documents.

• Types:

o Binary classification (spam vs. not spam),

o Multi-class classification (classify news into politics, sports, tech),

o Multi-label classification (a movie review may be both “comedy” and “romance”).

Use Case: Email spam detection, sentiment analysis (positive, neutral, negative).
3. Clustering

• Goal: Group similar documents together without pre-defined labels.

• Based on similarity metrics (e.g., cosine similarity, Jaccard).

• Unsupervised learning method.

Use Case: Grouping customer reviews to find emerging themes or topics.

4. Topic Modelling

• Goal: Discover abstract “topics” hidden in a collection of documents.

• Uses algorithms like:

o LDA (Latent Dirichlet Allocation)

o NMF (Non-negative Matrix Factorization)

Use Case: Automatically identifying topics people are discussing in online forums.

5. Summarization

• Goal: Generate a concise summary of a document while preserving key information.

• Types:

o Extractive: Picks key sentences from the text.

o Abstractive: Generates new sentences to summarize (like how humans write summaries).

Use Case: Summarizing lengthy legal or research documents.

6. Sentiment Analysis

• Goal: Detect the emotional tone (opinion, attitude, mood) in text.

• Outputs:

o Polarity: Positive, Negative, Neutral.

o Emotion: Happy, Angry, Sad, etc. (more advanced).

• Uses NLP techniques and ML models.

Use Case: Monitoring brand sentiment on Twitter or customer reviews.

7. Language Modelling / Text Generation

• Goal: Predict the next word(s) in a sentence or generate entire paragraphs.

• Based on probabilities of word sequences (e.g., n-grams, neural networks).

 • Used in:

o Auto-complete

o Chatbots

o Machine Translation

Use Case: Suggesting next words while typing in Google search or writing an email.

Q.8. what is analytical sandbox and why it is important

Ans:- An analytical sandbox is a secure, isolated environment where data scientists and analysts can explore,
analyse, and experiment with data without affecting the live production systems or databases.

Key Features:

• Isolated environment: Separate from production, so mistakes don’t impact business operations.

• Access to curated data: Usually includes cleansed, structured, and often anonymized data.

• Flexible tools: Supports various analytics tools, programming languages (like Python, R, SQL), and
libraries.

• Scalability: Often built on scalable infrastructure (cloud-based or on-premise clusters).

Why It's Important:

1. Safe experimentation:
Analysts can test hypotheses, models, and scripts freely without the risk of disrupting live systems.

2. Accelerates innovation:
Speeds up development of data-driven insights, models, and dashboards by providing a flexible
and independent space.

3. Supports reproducibility:
Workflows and experiments in a sandbox are easier to document and repeat, supporting
transparency and auditing.

4. Improves collaboration:
Multiple users can share the same controlled environment, making it easier to collaborate on
analytics projects.

5. Governance and security:

Organizations can control access, ensure data privacy, and enforce compliance while still
empowering analysts.
Q.9 . How does ARMA model differ from ARIMA model? When ARMA is more suitable

Ans :- The main difference between ARMA and ARIMA models lies in their ability to handle non-stationary data.
ARMA models are suitable for stationary time series, while ARIMA models can also handle non-stationary data
by incorporating an "integration" component that transforms the data into stationarity through differencing. [1,
2]

ARMA Model:

• Assumes stationarity: ARMA models assume that the time series data has a constant mean and
variance over time. [1, 3]

• Suitable for stationary data: When the data is already stationary, ARMA models can effectively
capture the relationships between past values and past errors to predict future values. [1, 4]

• Notation: ARMA(p, q), where 'p' is the order of the autoregressive (AR) component and 'q' is the
order of the moving average (MA) component. [4, 5]

ARIMA Model: [1, 1, 2, 2]

• Handles non-stationarity: ARIMA models can handle time series data that is not stationary by
applying a differencing process to transform it into stationarity. [1, 1, 2, 2]

• Integration component: The 'integrated' part in ARIMA refers to the order of differencing
(denoted by 'd' in ARIMA(p,d,q)) required to make the data stationary. [1, 1, 5, 5]

• Suitable for non-stationary data: When the data has trends, seasonality, or other non-stationary
characteristics, ARIMA models can effectively capture these patterns by first removing the non-
stationarity through differencing and then modeling the remaining stationary data. [1, 1, 6, 6, 7]

Q.10. Write short note on box Jenkins Methodology?

Ans : - Short Note on Box-Jenkins Methodology

The Box-Jenkins Methodology is a structured approach to modeling time series data, particularly using ARIMA
models (AutoRegressive Integrated Moving Average). Developed by George Box and Gwilym Jenkins, it focuses
on identifying, estimating, and validating models for forecasting.

Key Steps in Box-Jenkins Methodology

1. Identification

o Determine if the series is stationary (constant mean and variance over time).

o Use plots, ACF (Autocorrelation Function) and PACF (Partial Autocorrelation Function) to
decide the ARIMA model order:

▪ AR (p), I (d), MA (q)

2. Estimation

o Fit the selected ARIMA model using statistical software.

 o Estimate the model coefficients using techniques like maximum likelihood estimation
(MLE).

3. Diagnostic Checking

o Check residuals (errors) to ensure they resemble white noise (no autocorrelation).

o Use Ljung-Box test, ACF plots, and residual analysis.

4. Forecasting

o Once the model passes diagnostic checks, use it to predict future values.

Applications

• Stock market prediction

• Sales forecasting

• Weather data modeling

• Traffic and energy demand analysis

Advantages

• Systematic and thorough

• Handles trends and seasonality with modifications (e.g., SARIMA)

• Widely supported in tools like R, Python, and Excel

Would you like a simple diagram or visual to summarize the steps?

Q. Explain different types of data visualisation in Python programming language?

Ans: - In today’s world, a lot of data is being generated on a daily basis. And sometimes to analyse this data for
certain trends, patterns may become difficult if the data is in its raw format. To overcome this data visualization
comes into play. Data visualization provides a good, organized pictorial representation of the data which makes it
easier to understand, observe, analyse. In this tutorial, we will discuss how to visualize data using Python.

Python provides various libraries that come with different features for visualizing data. All these libraries come
with different features and can support various types of graphs. In this tutorial, we will be discussing four such
libraries.

• Matplotlib

• Seaborn

• Bokeh

• Plotly
 Matplotlib
• Matplotlib is an easy-to-use, low-level data visualization library
that is built on NumPy arrays. It consists of various plots like
scatter plot, line plot, histogram, etc. Matplotlib provides a lot of
flexibility.
Scatter Plot
• Scatter plots are used to observe relationships between
variables and uses dots to represent the relationship between
them. The scatter() method in the matplotlib library is used to
draw a scatter plot.

•
• This graph can be more meaningful if we can add colors and also
change the size of the points. We can do this by using the c and
s parameter respectively of the scatter function. We can also
show the color bar using the colorbar() method.

Line Chart

Line Chart is used to represent a relationship between two data X and

Y on a different axis. It is plotted using the plot() function. Let’s see the
below example.
Bar Chart

A bar plot or bar chart is a graph that represents the category of data
with rectangular bars with lengths and heights that is proportional to
the values which they represent. It can be created using
the bar() method

Histogram

A histogram is basically used to represent data in the form of some

groups. It is a type of bar plot where the X-axis represents the bin
 ranges while the Y-axis gives information about frequency.
The hist() function is used to compute and create a histogram. In
histogram, if we pass categorical data then it will automatically
compute the frequency of that data i.e. how often each value occurred.

•
•

Q. Explain different types of data visualizations in R programming language.

Ans:- https://www.geeksforgeeks.org/data-visualization-in-r/

Q. How Exploratory Data Analysis (EDA) is performed in R?

Ans:-

Exploratory Data Analysis or EDA is a statistical approach or technique for analyzing data sets to summarize
their important and main characteristics generally by using some visual aids. The EDA approach can be used to
gather knowledge about the following aspects of data.

• Main characteristics or features of the data.

• The variables and their relationships.

• Finding out the important variables that can be used in our problem.

EDA is an iterative approach that includes:

• Generating questions about our data

• Searching for the answers by using visualization, transformation, and modeling of our data.

• Using the lessons that we learn to refine our set of questions or to generate a new set of
questions.

Exploratory Data Analysis in R

In R Programming Language, we are going to perform EDA under two broad classifications:

• Descriptive Statistics, which includes mean, median, mode, inter-quartile range, and so on.

• Graphical Methods, which includes histogram, density estimation, box plots, and so on.
Before we start working with EDA, we must perform the data inspection properly. Here in our analysis, we will be
using the loafercreek from the soilDB package in R. We are going to inspect our data in order to find all the typos
and blatant errors. Further EDA can be used to determine and identify the outliers and perform the required
statistical analysis. For performing the EDA, we will have to install and load the following packages:

• “aqp” package

• “ggplot2” package

• “soilDB” package

We can install these packages from the R console using the install.packages() command and load them into our
R Script by using the library() command. We will now see how to inspect our data and remove the typos and
blatant errors.

Data Inspection for Exploratory Analysis in R

To ensure that we are dealing with the right information we need a clear view of your data at every stage of the
transformation process. Data Inspection is the act of viewing data for verification and debugging purposes,
before, during, or after a translation. Now let’s see how to inspect and remove the errors and typos from the
data.

Descriptive Statistics Exploratory Data Analysis in R

For Descriptive Statistics in order to perform EDA in R, we will divide all the functions into the following
categories:

• Measures of central tendency

• Measures of dispersion

• Correlation

We will try to determine the mid-point values using the functions under the Measures of Central tendency.
Under this section, we will be calculating the mean, median, mode, and frequencies.

Graphical Method in Exploratory Data Analysis in R

Since we have already checked our data for missing values, blatant errors, and typos, we can now examine our
data graphically in order to perform EDA. We will see the graphical representation under the following
categories:

• Distributions

• Scatter and Line plot

Under the Distribution, we shall examine our data using the bar plot, Histogram, Density curve, box plots, and
QQplot.

Q. Enlist and explain the steps of text analysis

Ans:- Steps of Text Analysis

Text analysis (also known as text mining or text analytics) involves extracting meaningful information from
unstructured text data. It is widely used in fields like social media analytics, sentiment analysis, customer
feedback evaluation, and more.

Here are the main steps of text analysis, with explanations:

1. Text Data Collection

• Description: Gather textual data from sources like documents, websites, tweets, customer
reviews, emails, etc.

• Tools: Manual entry, web scraping (e.g., using rvest in R or BeautifulSoup in Python), APIs (Twitter
API, etc.).

2. Text Preprocessing

• Description: Clean and normalize text data to make it suitable for analysis.

• Common Steps:

o Lowercasing text

o Removing punctuation

o Removing numbers

o Removing stopwords (e.g., "the", "is", "in")

o Tokenization (splitting text into words or tokens)

o Stemming (e.g., "running" → "run") or Lemmatization

3. Text Representation

• Description: Convert cleaned text into a numerical format for analysis.

• Common Methods:

o Bag of Words (BoW): Counts word frequency.

o Term Frequency–Inverse Document Frequency (TF-IDF): Weights words by importance.

o Word Embeddings (advanced): e.g., Word2Vec, GloVe.

4. Exploratory Text Analysis

• Description: Explore word frequencies, co-occurrence, and structure of the text.

• Examples:

o Word frequency table

 o Word cloud visualization

o Bar plots of most common terms

5. Sentiment Analysis

• Description: Analyze the emotional tone (positive, negative, neutral) of the text.

• Methods:

o Lexicon-based (using sentiment dictionaries)

o Machine learning-based (train models to predict sentiment)

6. Topic Modeling

• Description: Identify abstract topics discussed in a collection of documents.

• Popular Algorithm: Latent Dirichlet Allocation (LDA)

7. Text Classification or Clustering

• Description:

o Classification: Assign categories (e.g., spam/not spam).

o Clustering: Group similar texts without pre-defined labels.

8. Interpretation and Visualization

• Description: Visualize insights from the text using charts, word clouds, network diagrams, etc.

• Tools: ggplot2, wordcloud, LDAvis (for topic modeling visualization)

9. Model Evaluation (for predictive tasks)

• Description: Evaluate the performance of sentiment or classification models.

• Metrics: Accuracy, Precision, Recall, F1-Score

Q. What is the difference between Pandas and Numpy

Ans :-