Mastering a GenAI transition: 


From PoC to production

reaktor.com
 INDEX

4.
Before PoC

8.
The PoC phase

12.
Transitioning from PoC to production

18. Beyond production

reaktor.com 02
INTRO

Not long ago, the buzz surrounding Generative AI (GenAI) was mostly 

about exploratory experiments and initial proofs of concept (PoCs). 

As technologists and business leaders began to grasp GenAI's potential 

for their businesses, we also started seeing a wider transition from PoCs 

to GenAI solutions in production.

While it’s exciting to see new AI projects start bearing their real-world fruits,
there is also a lot to consider. Entering this new phase in production brings up
an entirely new set of challenges. In this guide, we break down some of the
key things to consider when making this transition.

reaktor.com 03
Before PoC

Setting the stage for success

reaktor.com 04
 BEFORE POC

Identifying high-impact 

use cases

Identifying use cases where GenAI can dramatically you might think. Despite the excitement, AI champions
improve your business is the best way you can get often also face significant resistance. Some individuals
your org’s foot in the door when it comes to wider- remain skeptical about the effectiveness of GenAI in
scale adoption of GenAI.

real-world applications, which is why it’s so important 


to demonstrate real value to real problems from the 

High levels of hype usually result in many 
 get-go.

people experimenting somewhat aimlessly. 


While prototyping is great, selecting specific use 
 A way to do this could be to take a look at the data you
cases that have a tangible impact on your business already have readily available. Recognizing what data
might be more beneficial. 

you have, how good it might be for a specific

application, or using some synthetic data in case of
For one, pitching GenAI solutions internally at your sensitivity concerns are all ways in which you can easily
organization might not always be as easy of a sale as test out the effectiveness of a new GenAI use case.

reaktor.com 05
The discovery phase of a new
GenAI opportunity should include
everyone, not just the developers
or key decision makers.

reaktor.com 06
 BEFORE POC

A knowledge-sharing

ecosystem for building

GenAI capabilities

Fostering an environment of knowledge sharing and problem-solving process. Business can bring in

ideation between all teams ensures that stakeholders valuable information about the problems while

are aligned and excited about GenAI possibilities.

developers can figure out what solution sounds 


most feasible.

In most organizations, AI remains only in the hands of a

few. This creates unnecessary gaps between a couple This technology is developing so fast that 


of AI developers and the rest of the team. To set the hosting sessions for discussion, sharing findings, 


stage for success with your Gen AI plans, it’s best to and keeping up with recent innovations is a great 


build a workflow where everyone is included. 

way to make sure your organization stays ahead.

Business knows where the impact is, but that has to 


The discovery phase of a new GenAI opportunity 
 be matched with feasibility. Creating a playground in

should include everyone, not just the developers or 
 which to test assumptions is a good way to share

key decision makers. By working together with knowledge and come up with new ideas. When done

business folks, designers, and project managers from right, a strong knowledge-sharing ecosystem can 


the get-go, you are keeping everyone up to date on make all the difference with building GenAI capabilities

latest developments, and including them into the throughout your organization.

Information 2
The PoC phase

Laying foundations

reaktor.com 08
 POC PHASE

Collaborative
development of PoCs

Going for quick wins with non-sensitive data 
 PoCs utilizing GenAI is a completely different 

helps gain momentum and demonstrate value.

game. Often it takes just two days to create 


one. The speed of development cycles is so much
PoCs are, above all else, about experimentation. 
 faster, and the nature of challenges developers face 

With non AI applications, PoCs require interviewing are also unique. When making a PoC is that fast, 

end-users, iterating in small cycles, and adopting rapid the question becomes: what will actually work in
iterative product development processes as much as production?

possible. These processes can take weeks or months,

as the gap between developer and end-user is usually The focus shifts to understanding what experiments
quite large. should be killed and which ones to double down on.

reaktor.com 09
When teams are able to
operate with high confidence
and autonomy, they have the
best opportunity for impact.

reaktor.com 10
 POC PHASE

Incorporating iterative
feedback loops
User

Query

Moderation Feedback system

Awnser Refined question

Dispatcher Unclear Intention capture bot

Wants to buy a product Needs help Needs extra services

Application Layer

Product finder bot Customer service bot Upselling bot

Incorporating real user feedback early and 
 To make this work, it’s important to facilitate tightly 

often to refine and align the PoC with actual 
 knit teams that embody high levels of trust. When
business needs.

teams are able to operate with high confidence and

autonomy, they have the best opportunity for impact.
When PoCs are so easy to create, it makes 
 Removing any possible red tape and enabling teams to
sense to split teams into even smaller ones. Running work like mini startups within your org allows them to
ever-smaller experiments that are fast and iterative 
 iterate on a PoC and generate value for actual business
can open up new possibilities for innovation. needs faster.

reaktor.com 10
Transitioning from PoC to production

Reaping the benefits

reaktor.com 12
 PRODUCTION

Overcoming LLM 

testing challenges
Retrieval

Correct context

Query

Evaluate retrieval Wrong context

Original sources and feedback loops help 
 It’s also important to note that organizations and
navigate the complexities of LLM testing.

applications have different tolerance levels for errors. Retrieved context

For example, a governmental org can’t give out false
When developing a new GenAI application, it’s not information under any circumstances, whereas an e- Not found

uncommon to get 80% of work done in just two 
 shop recommendation system suggesting a product
days. These prototypes can not only look impressive, 
 that is not 100% a match won’t do any major harm.

but also function effectively in various scenarios. 


These AI applications often utilize Retrieval 
 By nature, AI systems might provide different 
 Augmented generation
Augmented Generation (RAG) – a technique that outputs for the same query – a fundamental deviation
Well formatted answer
leverages a specific data source within the company, from traditional software development. To address this,
such as internal support articles. When a customer 
 developers have to create sophisticated evaluation Prompt
asks a question, the system retrieves all relevant 
 systems, such as incorporating simple feedback
articles which are then compiled into a 
 mechanisms (like up/down votes in ChatGPT or more Evaluate answer Incoherent answer

prompt and sent to an LLM.

complex rating systems on scales of 1-5, for example). Answer/Output

These systems not only guide the AI in improving
However, developing AI apps comes with its 
 responses but also enable pointing users back to Missing information

own set of unique issues. For example, ensuring a original sources, such as support articles. 

customer service bot doesn’t inadvertently mention

competing products, even if only in 1/100 cases. The feedback loop plays a crucial role, essentially

Sometimes, the model ‘hallucinates’, generating for-ming a dataset of perfect questions and answers
responses that deviate significantly from the 
 that can be used to further refine the LLM’s
actual data. The challenge here is not just in crafting performance.
effective prompts and running thorough testing, but
actually being able to transition these applications to
production while ensuring consistent performance.
PRODUCTION

Strategic evaluation 

Understanding the strategic implications of 
 where the Mercedes Benz chatbot mistakenly 

moving to production, including both tech and starts recom-mending a Tesla vehicle due to 

business impacts, is crucial in ensuring your 
 insufficient testing.

for production GenAI investment pays off.

Setting clear KPIs for what needs to be achieved when

Like mentioned in the section above, 

establishing a feedback loop is the next important 

rolling out to production is essential. It’s often safer to step for continuous improvement. In addition to human
roll out a product in phases, rather than releasing evaluation mechanics, automating improvements can
updates to the entire customer base simultaneously. bring another perspective to the process. One bot 

This approach helps mitigate risk, and avoid situations monitoring the main bot, and so on.

Level 0
 Level 1
 Level 2
 Level 3
 Level 4


Unaware Exploration Proof-of-Concepts Production use Platformization

GenAI enabled
 E xploration and First GenAI First GenAI services Solid data pipelines and platforms
services discovery process concepts and PoCs in production for scaling GenAI services

ChatGPT and Copilot Forming organizational Provenly improving Systematic 


Professional use
sporadic use polices for GenAI use professional work organizational learning

reaktor.com 14
LLMs will inevitably hallucinate
every so often, so you’ll want to
prevent this with rigorous testing
and training.

reaktor.com 15
 PRODUCTION

Preventing hallucinations
User question
and ensuring reliability
LLM

Output

Answer 1 Answer 2 Answer 3

Output

LLM evaluator
Implementing different strategies to prevent data correctly, it’s wrong. It should only be able to 

inaccuracies is your best bet for making sure your answer what is available in your organization’s
machine doesn’t start to hallucinate.

knowledge base.

LLMs will inevitably hallucinate every so often, 
 When the tech powering space shuttle was flaky, the
so you’ll want to prevent this with rigorous testing and engineers implemented a four-computer check to see 

Most correct answer training. LLMs have what we call “world knowledge”, if things were going right. This same principle can be
and you are teaching them your knowledge, or in this used with LLMs. One model checks whether things are
case the knowledge of your company. In order to test consistent, and others need to agree on the answer.
whether your systems are accurately making this This can work with certain tests, searching for
transition, you can test it by asking simple questions
keywords, or asking questions that can only have 

such as “Who is the President?”. If the model answers exact answers.

reaktor.com 16
 PRODUCTION

Ensuring seamless
integration across all
existing systems

A successful push into production needs 
 configurations, who to contact if something 


to take into consideration the inter-dependencies breaks, and who is responsible for fixes?

across your existing systems.

Pushing a successful PoC fully into production 


It’s also important to note that the work needed to and integrated across all systems and processes 

integrate a new AI system into existing IT infrastructure can require an order of magnitude of more work. 

can be a noteworthy challenge. Things to consider In production, no component lives in isolation. 

include how it affects existing processes and workflows, And most importantly, new components must not 

who maintains it, who needs to be aware of new break any existing pieces.

reaktor.com 16
Beyond production


Staying ahead of competition

reaktor.com 18
The goal shouldn’t be to just
adopt new technology, but to
strategically position your
business for the future.

reaktor.com 19
 BEYOND PRODUCTION

Continuous improvement 

and innovation

Constant innovation to ensure your GenAI 
 with just how fast things are developing. Having a
solutions evolve with your business will help 
 dedicated team to keep up with the tech and spread
you stay ahead of competition.

knowledge can be a great way to stay ahead of the

curve and make sure you’re up to date on the latest
There is a lot we can learn from “traditional AI” in developments. That said, integrating GenAI holistically
production, and particularly MLOps. Things such as across your organization is the only sure way to solve
how you can keep models retrained, updated, and business challenges and get results, while staying
monitored as a part of normal development and mindful of feasibility.

DevOps work. 

The goal shouldn’t be to just adopt new technology, 


That being said, LLMs move at incredible speeds, 
 but to strategically position your business for the future
and most companies are not agile enough to keep up – leveraging GenAI for real competitive advantage.

reaktor.com 20
 ABOUT

We’re a global technology consultancy that

combines strategy, design, development and
creative to help today’s forward-thinking
companies become tomorrow’s digital leaders.
More about our services: 

reaktor.com/services

reaktor.com