Assignment #2

course code 8510

course name Business research methods
Topic number 9
Topic special data problems and their
solutions.

Here’s a look at some common data problems and how you can solve them.

Coming Up

What is Big Data?

1. Finding and Fixing Data Quality Issues

2. Long Response Times from System

3. Confusion with Big Data Tool Selection

4. Real Time Big Data Problems

5. Lack of Understanding

6. High Cost of Data Solutions

7. Too Many Choices

8. Complex Systems for Managing Data

9. Security Gaps

10. Low Quality and Inaccurate Data

11. Compliance Hurdles

12. Using Data for Meaning

13. Keeping Up With Growth in Data

14. Accessibility

15. Pace of Technology

16. Lack of Skilled Workers

17. Data Integration

18. Processing Large Data Sets

19. Constantly Changing Data

What is Big Data?

A commonly used buzzword, big data is everywhere, which is why it’s always being talked about!
Data is collected and generated from almost everything we do- be it streaming a show,
submitting a form online, sending an email, reading a text, creating a report, etc.

To define what big data is, the four V’s are used, namely:

Volume: The size of data (i.e. petabytes or exabytes)

Velocity: The speed at which data flows

Veracity: The validity of the data

Variety: The nature of the data (structured and unstructured formats)

1. Finding and Fixing Data Quality Issues

When dealing with the data, the utmost importance is its accuracy. After all, every insight you
glean from data will depend on the data itself. It all begins during the data collection phase. At
this time, you want to be sure that you’re collecting data from the right sources at the right
time if you’re going to apply the data for outputs.
Along with the collection of data, data quality will also depend on how you store the data. It
must be made accessible in order to be analysed (this is where automation solutions come into
play).

During the data lifecycle, you must also maintain data properly so that it can be used by the
right team at any point in time for application. This data usage is what breeds decision-making
abilities. To learn how to keep data clean and reduce data quality issues, check out this guide.

2. Long Response Times from System

Clean and accurate data is just as important as data being accessible when you need it. If you’re
using a data tool that’s slow, then by the time your data is available for use, it could be
considered outdated and old.

You want data to be able to be processed immediately when you input it so you can make use
of your output in a timely manner.

One way to fix long response times from your system is to ensure that data is being stored
efficiently by performing data re-engineering. Or, look for a more optimised data system that’s
scalable for your growing data needs.

3. Confusion with Big Data Tool Selection

Another challenge when dealing with big data is choosing the right big data tool for your
business’ needs. Since your big data tool is geared towards reducing big data problems, it’d be a
shame if it became a problem in itself!

In order to overcome this challenge, it’s best to take time performing research and not jump
too quickly into a specific tool. Additionally, be sure to review what kind of support the tools
you’re considering offer.

If the option exists to schedule a demo, take advantage of it because it will give you a view of
how the big data solution will work specifically for your business.

4. Real Time Big Data Problems

As we mentioned, big data exists everywhere. As such, data is constantly changing and evolving,
which thus impacts the insights you glean from it. Technically, this requires a tool that can
provide up-to-date filtering and remove redundant or irrelevant data from the picture when
you’re applying it.

A surefire way to overcome real-time big data issues is to deploy an automation solution that
utilises artificial intelligence (AI) to process, analyse, and structure data in real-time. By doing so,
you can avoid big data problems at every turn.

5. Lack of Understanding
Companies can leverage data to boost performance in many areas. Some of the best use cases
for data are to: decrease expenses, create innovation, launch new products, grow the bottom
line, and increase efficiency, to name a few. Despite the benefits, companies have been slow to
adopt data technology or put a plan in place for how to create a data-centric culture. In fact,
according to a Gartner study, out of 196 companies surveyed, 91% say they have yet to reach a
“transformational” level of maturity in their data and analytics.

Solution: One way to combat the slow adoption is to take a top-down approach for introducing
and training your organisation on data usage and procedures. If your in-house team doesn’t
have the resources to take this on, consider bringing in IT specialists or consultants and holding
workshops to educate your organisation.

6. High Cost of Data Solutions

After understanding how your business will benefit most from implementing data solutions,
you’re likely to find that buying and maintaining the necessary components can be expensive.
Along with hardware like servers and storage to software, there also comes the cost of human
resources and time.

Solution: To make the most informed decision for what kind of data solution will provide the
most ROI, first consider how and why you want to use data. Then, align your reasoning with
your business goals, conduct research for available solutions, and implement a strategic plan to
incorporate it into your organisation.

7. Too Many Choices

According to psychologist Barry Schwartz, less really can be more. Coined as the “paradox of
choice,” Schwartz explains how option overload can cause inaction on behalf of a buyer.
Instead, by limiting a consumer’s choices, anxiety and stress can be lessened. In the world of
data and data tools, the options are almost as widespread as the data itself, so it is
understandably overwhelming when deciding the solution that’s right for your business,
especially when it will likely affect all departments and hopefully be a long-term strategy.

Solution: Like understanding data, a good solution is to leverage the experience of your in-
house expert, perhaps a CTO. If that’s not an option, hire a consultancy firm to assist in the
decision-making process. Use the internet and forums to source valuable information and ask
questions.

8. Complex Systems for Managing Data

Moving from a legacy data management system and integrating a new solution comes as a
challenge in itself. Furthermore, with data coming from multiple sources, and IT teams creating
their own data while managing data, systems can become complex quickly.

Solution: Find a solution with a single command center, implement automation whenever
possible, and ensure that it can be remotely accessed 24/7.

9. Security Gaps
The importance of data security cannot go unnoticed. However, as solutions are being
implemented, it’s not always easy to focus on data security with many moving pieces. Data also
needs to be stored properly, which starts with encryption and constant backups.

Solution: You can take a few low effort steps to dramatically increase the security of your data,
like: automate security updates, automate backups, install operating system updates (which
often include better security), use firewalls, etc.

10. Low Quality and Inaccurate Data

Having data is only useful when it’s accurate. Low quality data not only serves no purpose, but
it also uses unnecessary storage and can harm the ability to gather insights from clean data.

A few ways that data can be considered low quality is:

Inconsistent formatting (which will take time to correct and can happen when the same
elements are spelled differently like “US” versus “U.S.”),
Missing data (i.e. a first name or email address is missing from a database of contacts),

Inaccurate data (i.e. it’s just not the right information or the data has not be updated).

Duplicate data (i.e. the data is being double counted)

If data is not maintained or recorded properly, it’s just like not having the data in the first place.

Solution: Begin by defining the necessary data you want to collect (again, align the information
needed to the business goal). Cleanse data regularly and when it is collected from different
sources, organise and normalise it before uploading it into any tool for analysis. Once you have
your data uniform and cleansed, you can segment it for better analysis.

11. Compliance Hurdles

When collecting information, security and government regulations come into play. With the
somewhat recent introduction of the General Data Protection Regulation (GDPR), it’s even
more important to understand the necessary requirements for data collection and protection,
as well as the implications of failing to adhere. Companies have to be compliant and careful in
how they use data to segment customers for example deciding which customer to prioritise or
focus on. This means that the data must: be a representative sample of consumers, algorithms
must prioritise fairness, there is an understanding of inherent bias in data, and Big Data
outcomes have to be checked against traditionally applied statistical practices.

Solution: The only solution to adhere to compliance and regulation is to be informed and well-
educated on the topic. There’s no way around it other than learning because in this case,
ignorance is most certainly not bliss as it carries both financial and reputational risk to your
business. If you are unsure of any regulations or compliance you should consult expert legal and
accounting firms specialising in those rules.

12. Using Data for Meaning

You may have the data. It’s clean, accurate and organised. But, how do you use it to provide
valuable insights to improve your business? Many organisations are turning to robust data
analysis tools that can help assess the big picture, as well as break down the data into
meaningful bits of information that can then be transformed into actionable outcomes.
Solution: Whether this means having a consistent reporting structure or a dedicated analytics
team, be sure to turn your data into measurable outcomes. This means taking data and
transforming into actions for the business to take in an effort to produce wins for the company.

13. Keeping Up with Growth in Data

Like scaling a company, growing with data is a challenge. You want to make sure that you can
scale your solution with the companies growth so that the costs and quality don’t decrease as it
expands.

Solution: This is achievable by creating projections from the get go of introducing data and data
management tools. Make sure that you select a robust data solution and know in advance that
it can handle the capabilities you may need down the line. Another option is to rely on support
systems and internal teams to manage aspects of growth. For example, you can define
milestones for your team to be aware of so that only when you reach them will you consider
moving to a more sophisticated system.

14. Accessibility
Sometimes, companies silo data to one person or one department. Not only does this put
immense responsibility on a select few, but it also creates a lack of accessibility throughout the
organisation in departments where the data can be of use to provide a positive impact. Data
silos directly inhibit the benefits of collecting data in the first place.

Solution: It sounds simple, but it’s not done enough - integrate your data. Set clear expectations
and create a unified system that can handle each department’s needs. If it’s not through finding
a single integrated system, consider using APIs so that data is accessible in one, centralised
location.

15. Pace of Technology

Inventor, author and futurist Ray Kurzweil put it best when he defined the accelerating rate of
change of technology. Each subsequent technological advancement builds more quickly upon
the last because they evolve at each step to become more efficient and therefore can better
inform what comes next. For example, just consider how rapidly cloud computing and artificial
intelligence are improving.
With the rapid advancement of technology and systems, you don’t want your data tools to
become outdated, especially when you’re investing time, energy and human resources into
them.

Solution: While you can’t stop progression, you can prepare for it. This begins with staying
informed of information technology and its new features, products and threats.

16. Lack of Skilled Workers

While the technological demand is high and artificial intelligence and data analysis tools are
innovating swiftly, the lack of skilled workers is causing a bottleneck for many companies. The
number of new, skilled graduates isn’t keeping pace with technology, and in turn, companies
are asking staff to supplement this shortfall by working multiple roles.

Solution: If the solution doesn’t exist naturally, try to create it. While you can’t control how
many data scientists and data analysts graduate each year, you can leverage your current
workforce

and provide training to instil and teach the skills you need them to have. You can also look for
more powerful data tools that make the analysis work less complex, which open up recruitment
to a broader pool of less specialised analysts.

17. Data Integration

Data integration consists of taking data from various sources and combining it to create
valuable and usable information.

Solution: There are a few ways to go about integrating data, including the following approaches:

Consolidation: Combining the data from various sources in one consolidated data store

Propagation: Leveraging applications to copy data from one location to another

Federation: Using a virtual database to create a model to match data from different systems

Virtualisation: Viewing data in one location, but where the data is still stored separately

18. Processing Large Data Sets

Large data sets are challenging to process and make sense of. The three V’s of big data include
volume, velocity and variety. Volume is the amount of data, velocity is the rate that new data is
created, and variety is the various formats that data exists in like images, videos and text.
Solution: The solution for problems with large data sets, regardless of their exact size, has been
discussed throughout this article and include tactics that are performed by both human
resources and technology. Steps to properly process data, regardless of its size, include
ensuring data is accurate, integrating data, and developing a company culture that both
understands and celebrates the usage of big data to make informed decisions.

19. Constantly Changing Data

Implementing the infrastructure and management of data cannot be a set-and-forget task. The
nature of data is that it’s constantly changing. Your customers details and orders are always
changing, as well as their interactions with your company.

Solution: Incorporate data systems with advanced machine learning and interoperability in
order to adapt to the constantly changing landscape of data inputs, and in turn, outputs. You
can also use systems that store historic as well as new data to understand the causes and
implications of the data changes and model future trends.

Stibo Systems - The Master Data Management Company

search

How to Improve Your Data Management

Search blog posts...

by Sabine Schmidbaur | March 31 2021 | 4 minute read

Five steps to unlock the value of your data and achieve compliance, using approved data
management principles

As your company’s connected and online business activities increase, so does the data
generated by these interactions. But, how are you using this data? Are you harnessing the
insights it imparts to increase sales or launch new products? If not, why not? This post gives you
5 steps on how to improve your data management efforts.

A study by the Global Association of Risk Professionals and SAS revealed that fewer than half of
the managers surveyed said their financial institution’s expertise on data management was
adequate or strong.
If you’re not quite there yet, it would appear on this basis that you’re not alone.

This lack of expertise is widespread across many sectors and industries, and it could be
depriving you of the opportunity to capitalize on the data you possess. In addition, it could put
you in risk of neglecting the protection of data, risking your brand reputation and regulatory
sanctions, such as the ones listed in the General Data Protection Regulation (GDPR) where fines
run up to 4% of annual turnover.

If you’re in the process of examining your organization’s data management position, these five
steps will help you improve your data management processes to the point of profitable impact.

Stibo Systems - The Master Data Management Company

search

How to Improve Your Data Management

Search blog posts...

by Sabine Schmidbaur | March 31 2021 | 4 minute read

Five steps to unlock the value of your data and achieve compliance, using approved data
management principles

As your company’s connected and online business activities increase, so does the data
generated by these interactions. But, how are you using this data? Are you harnessing the
insights it imparts to increase sales or launch new products? If not, why not? This post gives you
5 steps on how to improve your data management efforts.

A study by the Global Association of Risk Professionals and SAS revealed that fewer than half of
the managers surveyed said their financial institution’s expertise on data management was
adequate or strong.

If you’re not quite there yet, it would appear on this basis that you’re not alone.

This lack of expertise is widespread across many sectors and industries, and it could be
depriving you of the opportunity to capitalize on the data you possess. In addition, it could put
you in risk of neglecting the protection of data, risking your brand reputation and regulatory
sanctions, such as the ones listed in the General Data Protection Regulation (GDPR) where fines
run up to 4% of annual turnover.

If you’re in the process of examining your organization’s data management position, these five
steps will help you improve your data management processes to the point of profitable impact.

how to improve data management

Five steps to improve your data management

process and plan
Step 1) Take business ownership of data management
Many organizations are now employing Chief Data Officers (CDOs) to help them manage data
governance and the use of data as a business asset. And, with good reason: a survey by
Forrester Research ("Top Performers Appoint Chief Data Officers") found organizations with a
CDO were 70% more likely to reduce risk and better ensure compliance than an organization
without a CDO.

But still, a considerable number of enterprises—60.6% according to a survey from NewVantage

Partners ("The Chief Data Officer Dilemma")—identify others as the point of primary
responsibility for data strategy and results, or claim no single point of accountability. Often the
“others” are from operational IT and not from the business. This can often lead to an unclear
viewpoint on how the organizations data helps realize strategic value for the business.

So, the first step is to place data management responsibility where it makes the most sense. By
re-shaping your business approach to data strategy, and by appointing individuals to take
responsibility for implementing and governing it, you begin to take ownership of your data.

You’ll now have accountability, and with this comes procedures and processes to help govern
your usage of data to uncover intelligence and insights from it, allowing you to tap into its value
in order to use it as a key asset for delivering business success.
Step 2) Connect data sources
How do you collect, process, store and consume data across your business? The likelihood is
that as your organization has grown, data management has become decentralized with no real
management strategy in place to support it, and, to be frank, it’s become a bit of a nightmare
to manage.

Disparate systems need to be connected and data architecture standardized. Once you’ve
achieved this, it becomes easier to take business ownership of data management, enabling you
to align your use of data with your business strategy.

Organizations working with fragmented legacy systems are experiencing insurmountable gaps
when they try to retrieve and utilise their data. They’re also risking non-compliance with new
regulations as they can’t provide the transparency needed to show how their information was
acquired and where it will ultimately be used.

You can choose to link your existing systems, replace them with a new one, or opt for a
combination. The essential thing is that your data system(s) primarily support your business
strategy. Secondly, they should provide data coherence and visibility, as well as deduplication,
and use archive technologies to allow you to protect more, while storing only what you need. It
also needs to be flexible enough to support future growth.

Step 3) Manage your metadata

Metadata provides information on your primary data such as when it was acquired, created or
revised—plus its location and how it’s formatted.

Managing your metadata and creating a consistent business information infrastructure is vital
when it comes to referencing, accessing and consuming business data. Many organizations are
missing basic facts including who owns specific information, how important that information is
and how relevant it is to the business.

Managing your metadata will help you map and classify information and assess its value so you
can protect and prioritise data that’s valuable to you
Step 4) Plan how to deliver your data
Once your data is under control, how can you format and deliver it for optimum use across the
business? This is particularly important if your company operates multiple ERP and CRM
technologies and ecommerce platforms.

Data delivery is about putting relevant information into the right systems and into the hands of
the right people in the right way. As the GDPR and other data privacy regulations require,
accurate reporting is essential. Businesses will be required to provide detailed information on
the acquisition, management and protection of the data they possess.

You need to have systems in place that enable users to take intelligent business decisions based
on data-driven insight, for instance by using customer data reports to enhance existing
products and create new ones geared towards an identified customer need.

Step 5) Adopt consistent data governance policies

A data governance operating model will help bring roles, policies and processes together to
help your organization guide and enable ownership, accountability and performance of data.
Many organizations fall into the trap of thinking technology alone brings successful data
governance. However, data governance is much more about people - in terms of organizational
change management than it is about the technology. Of course technology can help enable
data governance in terms of providing analytics, insights etc., but only when a robust and
flexible data governance operating model has been defined and adopted into the business.

This will ensure that data is recognized as a valuable, strategic asset and its accountability is
clearly defined and implemented to follow internal and external regulations throughout the
data lifecycle.

________________________________________________