w LLMs, Semantic Kernel, Flowise, C#

jihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA

MAR
APR
2024

YEARS

Exploring AL
with Large
Language Models

Understanding Next-Level Dive into

High Microservices Distributed
Performance C# with Aspire Caching Solutions
1/76
 A R E Y O U W O N D E R IN G
H O W A R T IF IC IA L
IN T E L L IG E N C E C A N
i B E N E F IT Y O U T O D A Y ?
EXECUTIVE BRIEFINGS
Are you wondering how Al can help your business? Do you worry about privacy or regulatory issues stopping
you from using Al to its fullest? We have the answers! Our Executive Briefings provide guidance and
concrete advise that help decision makers move forward in this rapidly changing Age of Artificial Intelligence
and Copilots!

We will send an expert to your office to meet with you. You will receive:
1. An overview presentation of the current state of Artificial Intelligence.
2. How to use Al in your business while ensuring privacy of your and your clients' information.
3. A sample application built on your own HR documents - allowing your employees to query
those documents in English and cutting down the number of questions that you
and your HR group have to answer.
4. A roadmap for future use of Al catered to what you do.

AI-SEARCHABLE KNOWLEDGEBASE AND DOCUMENTS

A great first step into the world of Generative Artificial Intelligence, Large Language Models (LLMs),
and GPT is to create an Al that provides your staff or clients access to your institutional knowledge,
documentation, and data through an Al-searchable knowledgebase. We can help you implement a first
system in a matter of days in a fashion that is secure and individualized to each user. Your data remains
yours! Answers provided by the Al are grounded in your own information and is thus correct and applicable.

COPILOTS FOR YOUR OWN APPS

Applications without Copilots are now legacy!
But fear not! We can help you build Copilot features into your applications in a secure and integrated
fashion.

C O N T A C T U S T O D A Y F O R A F R E E C O N S U L T A T IO N A N D D E T A IL S A B O U T O U R S E R V IC E S .

c o d e m a g .c o m /ai-s e rv ice s
832-717-4445 ext. 9 • info@ codem ag.com
 TABLE OF CONTENTS

Features
8 CODE: 25 Years A g o 58 D is tr ib u te d C a c h in g :
CODE M a g a z in e h a s b e e n d o c u m e n t in g o u r i n d u s t r y 's c r e a t i v i t y
f o r a q u a r t e r o f a c e n t u r y . M a rk u s t a k e s a t r i p d o w n m e m o r y Lane
E n h a n c in g S c a la b ility a n d
t o s h o w u s h o w f a r w e 'v e c o m e . jihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA
P e rfo rm a n c e in ASP.NET 8 C o re
M a rk u s E g g e r
J o y d i p c o v e r s t h e p r o s a n d c o n s o f d i s t r i b u t e d c a c h in g a n d
t e a c h e s y o u h o w t o im p l e m e n t i t i n A S P .N E T C o re .
14 Passkey A u th e n tic a tio n J o y d ip K a n jila l

W e 're a l l L o o k in g f o r w a r d t o t h e d a y w h e n p a s s w o r d s a re n o L o n g e r
n e c e s s a r y . S a h il t a k e s a Lo o k a t h o w t o g e t y o u r a p p li c a t i o n s re a d y .
71 C# f o r H ig h -P e rfo rm a n c e S ys te m s
S a h il M a lik
I t t u r n s o u t t h a t y o u c a n w r i t e h ig h - p e r f o r m a n c e c o d e i n C# t h a t
m a tc h e s o r e x c e e d s t h e p e r f o r m a n c e o f n a t iv e c o d e . O re n s h o w s y o u
22 Y o u 're M is s in g O u t o n h o w t o r e t a in a l l t h e a d v a n ta g e s o f C# w h i l e y o u 'r e a t i t .

O ren E in i
O p e n -S o u rc e LLM s!
P h ili p p e x a m in e s L a rg e L a n g u a g e M o d e ls a n d C h a t-G P T ,
a n d h o w t o g e t t h e m t o d o w h a t YO U w a n t .

P h ilip p B a u e r

30 P r o to ty p in g L a n g C h a in
Departments
A p p lic a tio n s V is u a lly U s in g F lo w is e E d ito ria l
T h e L a n g C h a in f r a m e w o r k c a n h e lp y o u q u e r y Large a m o u n t s o f d a t a ,
a n d W e i- M e n g s h o w s y o u h o w t o u s e F lo w is e t o d o i t , e v e n i f y o u 'r e
n o t w r it in g co d e .
A d v e rtis e rs In d e x
W e i-M e n g L ee

C o d e C o m p ile rs
46 S e m a n tic K e rn e l 101: P a rt 2
M ic r o s o f t S e m a n tic K e r n e l is a n o p e n - s o u r c e A l f r a m e w o r k .
I t ' s v e r y n e w , a n d M ik e e x p lo r e s t h e c o d e t h a t h e lp s y o u u s e
th is p o w e r fu l n e w to o l.

M ik e Y e a g e r

52 A s p ira tio n a l .NET:

W h a t Is .NET A s p ire ?
S h a w n t a k e s t h e m is e r y o u t o f c o o r d i n a t i n g m ic r o s e r v ic e s
a n d d i s t r i b u t e d a p p li c a t i o n s u s in g A s p ir e , a c o o l n e w t o o l f r o m
t h e A S P .N E T t e a m a t M ic r o s o f t .

S h a w n W ild e r m u t h

US s u b s c r ip tio n s are US $ 2 9 .9 9 f o r o n e y e a r. S u b s c rip tio n s o u ts id e t h e US p a y

$ 5 0 .9 9 USD. P a y m e n ts s h o u ld be m a de in US d o lla rs d ra w n o n a US b a n k . A m e ric a n

CO DE^O
E xpress, M as te rC ard , V is a , a n d D is c o v e r c r e d it c ards are a c c e p te d . B a ck is s u e s are
a v a ila b le . For s u b s c r ip tio n in f o r m a tio n , s e n d e - m a il t o s u b s c r ip tio n s @ c o d e m a g .c o m
o r c o n t a c t C u s to m e r S e rv ic e a t 8 3 2 - 7 1 7 - 4 4 4 5 e x t. 9 .

YEARS S u b s c rib e o n lin e a t w w w .c o d e m a g .c o m

CODE C o m p o n e n t D e v e lo p e r M a g a z in e (ISSN # 1 5 4 7 - 5 1 6 6 ) is p u b lis h e d b im o n th ly

by EPS S o ftw a re C o rp o ra tio n , 6605 C yp re ss w oo d D riv e , S u ite 425, S p rin g , TX
7 7 3 7 9 U .S .A . POSTMASTER: S e nd a d d re s s c h a n g e s t o CODE C o m p o n e n t D e v e lo p e r
M a g a z in e , 6 6 0 5 C y p re s s w o o d D riv e , S u ite 4 2 5 , S p rin g , TX 7 7 3 7 9 U .S .A .

4 Table of Contents codem ag.com

LEADTOOLS

M o d u la r M e d ic a l W e b V ie w e r C o n tr o l

3V

tAeta d o t
°

_ o w -C o d e M o d u la r D esign 3D V olum e R en de rin g

W orks in a n y B row ser MRTI S u p p o rt Fast, C lie n t-S id e Tools

Im a g e M a rk u p a n d A n n o ta tio n DICO M a n d D IC O M w e b C o m p lia n t

G e t S ta rte d

DO W NLO AD OUR F R E E E V A L U A T IO N

L E A D T O O L S .C O M
 E D ITO R IA L

A re You Ready to Pivot?

As they like to say, a picture is worth a thousand words and the picture you see below (Figure 1) is
proof positive. This editorial you're reading is a direct result of the ideas I "chicken-scratched" on a stack
of post-it notes. This set of notes came from a conversation Melanie Spiller and I were having about
collaboration, in particular the concept of remote collaboration. We discussed at length how our work,
and how we perform it, was greatly transformed during the COVID crisis. COVID affected every one of
us in profound ways and we were forced to create new ways of working together, albeit remotely. We
had to change how we collaborated.

A Lot o f th e c o n v e rs a tio n s I have w ith M ela­ A long w ith being a darn fin e e dito r, Melanie is an J a m u lu s le ts y o u p la y , re h e a rs e , o r ja m w it h
n ie a b o u t c re a tiv ity and c o lla b o ra tio n c e n te r accom plished m usician and she collaborates w ith y o u r fr ie n d s , y o u r b a n d , o r a n y o n e y o u f in d
aro und m any n o n -te c h fie ld s . Every c re a tiv e numerous musicians in many d iffe re n t organiza­ o n lin e . P lay t o g e t h e r r e m o te ly in t im e w it h
en dea vor shares som e a com m on DNA and by tio ns . In norm al tim es, these collaborations take h ig h q u a lity , lo w -la te n c y so u n d on a n o rm a l
e x a m in in g o th e r fie ld s n o t re la te d to y o u r ow n, place in person. Rehearsals, practice, organiza­ b ro a d b a n d c o n n e c tio n .
y o u m ig h t fin d new er ways o f th in k in g . A t a t i o n - a l l done in person. COVID changed a ll o f
m in im u m , y o u 'll lik e ly learn s o m e th in g yo u th a t. I t d id n 't take to o long fo r these musicians to The key phrase in th is s ta te m e n t is Low-Latency.
d o n 't know. T ha t's a w in /w in , rig h t? realize th a t creating th e fid e lity o f in-person m u­ The fo lk s w ho cre ate d Jam ulus cre ate d a s o lu ­
sicianship was d iffic u lt to p u ll o ff rem otely and, tio n t h a t ena bled re m ote c o lla b o ra tio n s t h a t
W ell, th is p a rtic u la r co n v e rs a tio n was focused in th e b eg in n in g , im possible. The reason? T im ing. was as close to an in -p e rs o n ex perie nce as pos­
on one cre a tiv e fie ld : m usic. We ta lk e d a b o u t Melanie and her collaborators soon discovered th e s ib le . This gro up o f in tre p id m usicians fo u n d a
how COVID changed th e w ay m usicians were bane o f rem ote collaborations using ordinary to ols t o o l t h a t helped th e m to m ove fo rw a rd . They
fo rce d to change th e ir c o lla b o ra tiv e process such as Zoom: lag. Technology people are i n t i ­ soon a da pted th e ir process to Jam u lus's ra th e r
d u rin g th e pandem ic. m ately fa m ilia r w ith th e concept o f lag and there rig id re q uire m e nts w ith g re a t and s a tis fy in g
are a bunch o f d iffe re n t term s used to describe success.
Before COVID, I was ta k in g g u ita r lessons. My lag, such as response tim es, ping tim es, TTL, etc.
te a c h e r came to m y house tw ic e a m o n th fo r We strive to Lower lag to make our w ork perform The m usicians w e re n 't th e o n ly fo lk s th a t
an in -p e rs o n session w here he ta u g h t me th e b e tte r or to a t least to have th e appearance o f a da pted th e ir processes. My w ife and I are cu r­
fo u n d a tio n s o f g u ita r p la y in g , in c lu d in g scales, b e tte r perform ance. Progress bars anyone? re n tly b in g e -w a tc h in g a show on N e tflix ca lle d
chords, and, w h a t was p ro b a b ly th e m o s t d if ­ The B la c k lis t. The episode la s t n ig h t c o m p le te ly
f ic u lt a sp ec t fo r me, tim in g . Scales and chords M elanie and her m usician frie n d s c o u ld n 't paper b le w m y m in d . I t was th e season fin a le o f sea­
were p ro b a b ly th e ea sie st p a rt. W ith p ra c tic e , I ov er th es e Lag issues. I t w o u ld be d if f ic u lt to son 7. This episode was in th e process o f be in g
g o t b e tte r and b e tte r as o u r lessons progressed. create g re a t perfo rm an ces , le t alo n e s a tis fy in g film e d d u rin g th e w o rs t p a rt o f th e COVID crisis
I t was t h a t cursed tim in g t h a t caused me th e and p ro d u c tiv e rehearsals, i f th e y c o u ld n 't c o n ­ w hen e v e ry th in g was b e in g s h u t d ow n. The e p i­
m o s t a g ita . The m etro no m e was n o t m y frie n d t r o l th e issue o f lag. So th e y searched fo r a sode began w ith a p ro lo g u e o f c lip s from th e
and, to be h on es t, I never re a lly g o t th e hang s o lu tio n to c o n tro llin g o r e lim in a tin g lag. This cas t and crew w here th e y m e n tio n e d t h a t th e y
o f i t . Over and over, he b e a t i t in to me t h a t i f I is w here th e y fo u n d a p ro d u c t c a lle d Jam ulus . were h a lfw a y done s h o o tin g th e epis ode w hen
ever w a n te d to p la y w ith o th e r fo lk s , I needed th e y were fo rce d to s h u t do w n .
to g e t tim in g d ow n . I t was th is tim in g th in g The Ja m u lu s w e b s ite describes th e ir p ro d u c t as
t h a t was p a rt o f m y discu ssio n w ith M elanie. fo llo w s : B u t th e y d id n 't s h u t d ow n. They re tre a te d te m ­
p o ra rily and came up w ith a s o lu tio n . The pa rts
o f th e film t h a t w e re n 't s h o t Live-action were
a n im a te d . I t was a b it odd, b u t i t w o rk e d! As a
film geek, i t was an a d d itio n a l tr e a t fo r me to
see how a show is s h o t o u t o f c o n tin u ity us ing a
re a l-w o rld exam ple. C hanging how th e y w orked
a llo w e d th es e c o lla b o ra to rs to a cc om plis h w h a t
th e y p ro b a b ly th o u g h t was an im p o s s ib le ta s k .
As G unny S eargent Thom as H ig hw ay w o u ld say:
They overcam e, th e y a da pted .

T his is w h a t we do as hum an be in gs . We o ver­

com e o bs ta cles . We a d a p t and we m ove fo r ­
w ard. How have y o u adapted?

F ig u re 1 : The o rig in o f th e species

6 Editorial codem ag.com

 The Boring Database jihgfedcbaZYXWV

D i s t r ib u t e d C o u n te rs

S p a tia l

P red icta ble •••

/ / s e a rc h
fro m Employees
w here s e a rc h (N o te s , ’ u n iv e r s it y ')

A ll-in -O n e / / F i l t e r & S p a t ia l
fro m R e s ta u ra n ts
w h e n Type = 'C o ffe e S hop' and s p a t i a l . w i t h i n (
s p a t i a l . p o i n t( A d d r e s s . L o c a ti o n . L a t i t u d e ,
A d d re s s . L o c a ti o n . Longi t u d e ) ,

E ffo rtle s s )
s p a t i a l . c i r c l e ( 2 0 , 4 7 .6 2 3 4 7 3 , -1 2 2 .3 0 6 0 0 9 7 )

/ / P lo t
fro m Employees F u l l - T e x t S e a rc h
s e le c t t im e s e r ie s (
fr o n H e a rtR a te s je tw e e n '2 0 2 0 - 0 1 - 0 1 ' and
) ... a n d M o r e
/ / A g g re g a te
fro m O rd e rs
g ro u p by S h ip T o . c it y
s e le c t s h ip T o . c it y as s h i p t o c i t y , sui ( L in e s [ ] . Q u a n t it y ) CBA

" A f t e r y e a r s a f f i x i n g m is t a k e s t h a t o t h e r s d i d in S Q L , d e v e lo p in g a

w o r ld - r e n o w n a p p l i c a t i o n t o f i n d c o m m o n o n e s , I r e a liz e d t h a t t h e
O re n Eini
p r o b l e m is n o t w i t h p e o p le , b u t w i t h t e c h n o lo g y . C EO & F ounder

T h a t d a y I s t a r t e d R a v e n D B ."

r a v e n d b .n e t/tr y
O N LIN E Q U IC K ID 2403021

C O D E: 25 Years A g o
As we continue our celebration of "30 years of CODE Group," we travel forward five years from last issue's installment of "the
Antique CODE Show" and look at our industry, and the world in general, 25 years ago (or so). The late 90s were an interesting
time for our industry. Compared to five years earlier, the internet had rapidly become mainstream. Although hardly anyonejihgfedc

o u ts id e t h e s o ftw a r e in d u s tr y h a d h e a rd o f i t in 1993, t h e FDA in M a rch o f 1 9 9 8 . I f y o u 'r e a re a d e r w h o w a s a l­

h a r d ly a n y o n e h a d CBA
n o t h e a rd o f i t b y 1 9 9 8 . re a d y in t h e in d u s t r y b a c k in 1 9 9 8 , t h is d e v e lo p m e n t m a y
h a v e m o re s ig n ific a n c e f o r y o u t o d a y th a n y o u im a g in e d
N e ts c a p e w a s t h e d o m in a n t p la y e r in t h e b ro w s e r m a rk e t, b a ck th e n .
o n e o f t h e h o t t e s t are as o f s o ftw a r e d e v e lo p m e n t, b a c k
t h e n . So m u c h s o , t h a t t h is is g e n e r a lly t h o u g h t o f as t h e T h is w a s a ls o t h e h e y d a y o f t h e d o tc o m b u b b le . T h e i n ­
era o f " t h e g r e a t b ro w s e r w a r s ," w it h M ic r o s o ft p u s h in g t e r n e t h a d g ro w n r a p id ly , a n d e v e ry o n e w a n te d t o in v e s t
i n t o t h e m a r k e t w it h I n t e r n e t E x p lo re r a n d g o in g as f a r in th e w ild e s t id e a s . V e n tu re c a p it a l in v e s t m e n t w e n t
a s — g a s p — b u n d lin g t h e b ro w s e r d ir e c t ly w it h t h e o p e r a t­ th r o u g h t h e r o o f a n d so d id IP O s. V e ry fe w h a d s o lid id e a s
M a rk u s E gger in g s y s te m . A lth o u g h it 's u n t h in k a b le t o d a y t h a t a n y PC o f h o w a n y o f t h is w a s g o in g t o m ake a n y m oney, b u t

m e g g e r@ c o d e m a g .c o m o r d e v ic e w o u ld n 't s u p p o r t d ir e c t in t e r n e t a cc e ss , b a c k " t h e n e w e c o n o m y " d i d n 't b o t h e r m u c h w it h s u c h d e t a il.

th e n , th is w as e n o u g h to c a u s e 2 0 U .S . s ta te s t o file W h o c a re s , w h e n y o u c an p a r ty lik e i t 's 1 9 9 9 ? T h e NAS­
M arkus, th e dyn am ic a n t i t r u s t c h a rg e s a g a in s t M ic r o s o ft. T h e a im w a s t o " d e ­ DAQ s to c k e x c h a n g e g re w f iv e - f o ld in th o s e y e a rs , u n t i l i t
fo u n d e r o f CODE G roup and a l l b le w u p a n d i t p lu n g e d fr o m a p e a k o f $ 5 ,0 4 8 .6 2 o n
te r m in e w h e th e r t h e c o m p a n y 's b u n d lin g o f t h e b ro w s e r
CODE M agazine's p u b lis h e r,
w it h t h e OS c o n s t it u t e d an u n f a ir m o n o p o lis t ic p r a c tic e ." M arch 1 0 , 2 0 0 0 , d o w n t o $ 1 ,1 3 9 .9 0 o n O c to b e r 4 , 2 0 0 2
is a ce le b ra te d M ic ro s o ft
A lth o u g h i t w a s i n i t i a l l y r u le d in 2 0 0 0 t h a t M ic r o s o ft ha d ( w h ic h ta k e s us a lm o s t i n t o t h e tim e fr a m e o f t h e n e x t
RD (R e g io n a l D ire c to r) and
v io la te d a n t i t r u s t la w s a n d t h e c o m p a n y w a s o rd e re d t o a r t ic le ) . A m o n g t h e b ig g e s t a n d m o s t v is ib le lo s e rs o f
m u lti-a w a rd -w in n in g MVP
b e s p l i t i n t o t w o e n t it ie s , t h a t r u lin g w a s la t e r o v e r tu r n e d t h e d o tc o m e ra a re P e ts .c o m , B o o .c o m , K o z m o .c o m , a n d
(M o s t V aluable P rofes sion­
o n a p p e a l, a n d M ic r o s o ft w a s a llo w e d t o c o n tin u e o p e r a t­ eT oys. H o w e v e r, th e r e w e r e n 't o n ly lo s e rs . A m a z o n .c o m ,
a l). A p r o lific coder, he's
in g as a s in g le e n t i t y ( a lth o u g h u n d e r s o m e L im ita tio n s eB ay , G o o g le , a n d s e v e ra l o th e r s , a re n o t j u s t s t i l l a ro u n d
in flu e n c e d and ad vised to p
t h a t a ffe c te d t h e o r g a n iz a tio n f o r a lo n g t im e t o c o m e ). b u t h a v e g ro w n i n t o s o m e o f t h e la r g e s t a n d m o s t i n f l u ­
F ortune 500 com pa nie s
e n t ia l c o m p a n ie s o n t h e p la n e t. I g u e s s o u r id e a s w e r e n 't
and has been a M ic ro s o ft
c o n tra c to r, in c lu d in g on B u t i t w a s n 't a l l a b o u t N e ts c a p e a n d M ic r o s o ft. T he m u ­ a l l c ra z y in th o s e da y s a f t e r a ll.

th e V is u a l S tu d io te a m . s ic in d u s tr y w a s t u r n e d u p s id e d o w n d u e t o t h e e m e r­
G lo b a lly re c o gn ize d as g e n c e o f N a p s te r, a p e e r - to - p e e r m u s ic s h a r in g s y s te m A b s e n t fr o m t h e l i s t o f i n f l u e n t i a l c o m p a n ie s in t h e Late
a speaker and a u th o r, t h a t u n ra v e le d t h e b u s in e s s m o d e l o f t h e e n t ir e in d u s tr y . 9 0 s is A p p le . T e e te rin g o n t h e b r in k o f b a n k r u p tc y , S te v e
Markus's e x p e rtis e spans A t t h e h e ig h t o f i t s p o p u la r ity , N a p s te r h a d 8 0 m illio n J o b s h a d j u s t re tu rn e d t o A p p le in 1 9 9 7 . Even M ic r o s o ft
A r t if ic ia l In te llig e n c e ( A l) , r e g is te r e d use rs w h o h a p p ily s h a re d m u s ic f o r fre e (a n d in v e s te d $150 m illio n in to A p p le in A ugust o f 1997,
.NET, c lie n t-s id e w eb, and a r g u a b ly il le g a l ly ) w it h t h e i r p e e rs , u n t i l t h e s e rv ic e w a s w h ic h , in h in d s ig h t , w a s p r o b a b ly n o t in s ig n if ic a n t as a
c lo u d d e v e lo p m e n t, fo c u s ­ s h u t d o w n in 2 0 0 1 . A t t h e tim e , a r t is t s t h a t t o p p e d t h e f a c t o r in t h e s u r v iv a l o f t h e c o m p a n y , as i t h e lp e d s t a b i­
in g on user in te rfa c e s , c h a r ts in c lu d e d n a m e s lik e B r itn e y S p e a rs , t h e B a c k s tre e t liz e t h e fin a n c e s o f t h e c o m p a n y a n d a llo w e d i t t o s u b s e ­
p ro d u c tiv ity , and m a in ta in ­ B o y s , a n d E m in e m . I t 's h a rd t o s a y w h ic h s o n g s w e re t h e q u e n t ly g o o n t o d e v e lo p n e w p ro d u c ts s u c h as t h e iM a c
able system s. A w ay from m o s t s h a re d , b u t M a d o n n a a n d E lto n J o h n s o n g s w e re up a n d t h e iP o d . B u t t h a t w a s a l l s t i l l in t h e f u t u r e a n d A p p le
w o rk , he's an e n th u s ia s tic th e r e , a n d t h e a r t is t s w e re a ls o a m o n g t h e m o re o u ts p o ­ c le a r ly w a s n 't v e ry c o o l in 1 9 9 8 , e x c e p t f o r s o m e n ic h e
w in d s u rfe r, scuba div er,
ken a g a in s t t h e p r a c tic e o f p e e r - to - p e e r s h a r in g f o r fre e . m a rk e ts , s u c h as d e s k to p p u b lis h in g . (W h e n w e re le a s e d
ice hockey player, g o lfe r,
E v e n tu a lly , t h e M e ta llic a v s . N a p s te r la w s u it w a s t h e s t a r t t h e f i r s t is s u e s o f t h i s m a g a z in e , i t w a s a l l d o n e o n M a c s ).
and g lo b e tro tte r, w ho
o f t h e e n d f o r t h e s e rv ic e .
Loves g a m in g on PC o r
T a lk in g a b o u t CODE: As f a r as CODE G ro u p g o e s , w e w e re
Xbox d u rin g ra in y days.
T he Late 9 0 s w e re a ls o a g r e a t t im e f o r m o v ie g o e rs ( w h ic h a ro u n d as a s m a ll b u t g r o w in g c o n s u ltin g a n d t r a in in g o r ­
in c lu d e d m e b a c k t h e n ) . F ilm s s u c h as " T it a n ic " ( 1 9 9 7 ) , g a n iz a t io n . O u r h e a d q u a rte rs h a d a lre a d y m o v e d t o H o u s ­
" S a v in g P riv a te R y a n " ( 1 9 9 8 ) , a n d "T h e M a t r ix " ( 1 9 9 9 ) t o n , Texas. We d id a l o t o f c o n s u ltin g a n d t r a in in g a n d
w e re a m o n g t h e m o s t p o p u la r. I t 's h a rd t o b e lie v e t h a t w e re v e ry a c tiv e in W in d o w s a n d in te r n e t- b a s e d d e v e lo p ­
i t 's b e e n a q u a r te r c e n tu r y s in c e t h e i r re le a s e . A m o n g t h e m e n t a n d h a d s ta r te d a r c h it e c t in g s y s te m s t h a t w e r e n 't
m o s t im p o r t a n t s o ftw a r e c o m p a n ie s t h a t g o t e s ta b lis h e d j u s t o b je c t- o r ie n t e d , b u t w e w e re s t a r t in g t o t h i n k a b o u t
in t h e Late 9 0 s w a s G o o g le (S e p te m b e r o f 1 9 9 8 ) . A lth o u g h c o m p o n e n t- b a s e d m u lt i- t ie r e d s y s te m s , r a th e r t h a n j u s t
i t 's h a rd t o b e lie v e t h a t J a m e s C a m e ro n f in is h e d " T it a n ic " " n e t w o r k e n a b le d " s o ftw a r e . T h is w a s a b ig d e a l a t a t im e ,
m o re t h a n 2 5 y e a rs a g o , i t 's e v e n h a rd e r t o b e lie v e t h a t a n d u lt im a t e ly le a d t o t h e "C O D E" b ra n d , w h ic h o r ig in a lly
G o o g le is y o u n g e r t h a n t h is m o v ie , as i t 's d i f f i c u l t t o d a y s to o d f o r C o m p o n e n t D E v e lo p e r. CO D E M a g a z in e h a d n 't
t o im a g in e a w o r ld w i t h o u t G o o g le . p u b lis h e d i t s f i r s t is s u e y e t (b e c a u s e t h e o v e r a ll CODE
G ro u p o r g a n iz a tio n g o e s b a c k f u r t h e r t h a n t h e m a g a z in e ),
P o lit ic a lly s p e a k in g , tim e s w e re a b i t s im p le r, as o u r m a in b u t t h is is w h e n w e f i r s t h a d t h e id e a o f p u b lis h in g a
c o n c e rn w a s B il l C lin to n 's Love f o r c ig a rs a n d in te r n s . I n ­ d e v e lo p e r m a g a z in e t h a t d i d n 't j u s t fo c u s o n a s in g le
t e r n a t io n a lly , n u c le a r te s ts b y P a k is ta n a n d I n d ia w e re p r o g r a m m in g la n g u a g e (s u c h as V is u a l B a s ic P r o g r a m m e r 's
t h e h o t t o p ic . T h e G oo d F rid a y A g re e m e n t in A p r il o f 1 9 9 8 J o u r n a l , F o x P r o A d v is o r , o r C /C + + U s e r 's J o u r n a l ) b u t c o v ­
b r o u g h t an e n d t o t h e v io le n c e in N o rth e rn Ir e la n d . I n e re d s o ftw a r e d e v e lo p m e n t a t a Larger, m o re t e c h n o lo g y -
s c ie n c e a n d m e d ic in e , w e h a d n 't y e t c o m p le te ly d e c o d e d o r ie n te d le v e l. T he id e a w a s Laughed a t b y m a n y a t t h e
h u m a n DN A, b u t o n t h e u p s id e , V ia g ra w a s a p p ro v e d b y tim e , b u t has s e rv e d us v e ry w e ll e v e r s in c e .

C O D E : 25 Y ears A g o codem ag.com

cbaZYXWVUTSRQPONMLKJIHGFEDCBA

F ig u r e 1 : T he W in d o w s 9 8 D e s k to p w it h F ile M a n a g e r

T he T e c h n o lo g y Landscape g r e a t s te p to w a r d s im p lif y in g t h e w o r ld o f p e rip h e ra ls .

W in d o w s 9 8 w a s t h e o p e r a tin g s y s te m m a n y o f us used A l l o f a s u d d e n , i t b e c a m e a l o t e a s ie r t o use p r in te r s ,
a t t h e tim e . I t te n d s t o b e a s o m e w h a t less p r o m in e n t s c a n n e rs , a n d c a m e ra s . W in d o w s 9 8 h a d a ls o im p r o v e d
re le a s e o f W in d o w s , b e tw e e n t h e m a jo r m ile s to n e s t h a t s u p p o r t f o r d e v ic e s s u c h as s o u n d c a rd s , v id e o c a rd s (w e
w e re W in d o w s 9 5 e a rlie r, a n d t h e s u b s e q u e n t a n d e x c e l­ d i d n 't c a ll th e m GPUs b a c k t h e n ) , and m odem s (y e s ,
le n t W in d o w s XP. H o w e v e r, W in d o w s 9 8 w a s a v e ry s o lid w e w e re s t i l l d ia lin g u p t o t h e in t e r n e t a n d p a y in g h o r ­
v e rs io n o f W in d o w s t h a t im p r o v e d o n W in d o w s 9 5 in a re n d o u s p h o n e b ills ) . W in d o w s 9 8 w a s a ls o m u c h fa s te r
v a r ie t y o f w a y s . I n t e r n e t E x p lo re r 4 .0 s h ip p e d o u t o f t h e th a n W in d o w s 9 5 , w it h im p r o v e d m e m o ry m a n a g e m e n t
b o x a n d w a s a v e ry s o lid b ro w s e r t h a t s u p p o r te d (a n d a n d d is k c a c h in g . T h e G U I ( g r a p h ic a l u s e r in te r f a c e ) o f
p u s h e d fo r w a r d ) t h e L a te s t v e rs io n o f H TM L. I t 's e a s y t o W in d o w s 9 8 w a s s im ila r t o t h a t f i r s t re le a s e d in W in d o w s
f o r g e t t h a t M ic r o s o ft w a s in s t r u m e n t a l in m a k in g b ro w s ­ 9 5 ( F ig u r e 1 ) . T h a t m e a n s a Lot o f " b a t t le s h ip g r a y " a p ­
ers a n d H T M L m o re o f a re a l d e v e lo p m e n t e n v ir o n m e n t, p lic a tio n s , as w e ll as a S t a r t M en u t h a t w a s n 't s e a rc h a b le .
s u p p o r tin g J a v a S c r ip t, VB S c r ip t, a n d a DOM ( D o c u m e n t ALL in a ll, W in d o w s 9 8 w a s a g r e a t c h o ic e as an o p e r a tin g
O b je c t M o d e l), w h ic h a llo w e d p ro g ra m m e rs t o in t e r a c t s y s te m f o r a u s e r's PC.
w it h e le m e n ts o n t h e p a g e in w a y s t h a t w e re p r e v io u s ly
u n t h in k a b le f o r w h a t us e d t o b e a s im p le d o c u m e n t a n d As fa r as h a rd w a re is c o n c e rn e d , users h a r d ly e v e r h a d PCs
t e x t d is p la y m e c h a n is m w it h h y p e r lin k s . I n 1 9 9 8 , i f y o u w it h m o re t h a n 3 2 M B o f RAM, s p o r tin g an I n t e l P e n tiu m
w a n te d t o Look a t a m o d e rn w e b s ite , I n t e r n e t E x p lo re r p ro c e s s o r ( F ig u r e 2 ) . S p in n in g h a rd d is k s w it h 4G B o f
w a s a g o o d c h o ic e in d e e d , as N e ts c a p e h a d an in c r e a s ­ s to ra g e w as t h e n o rm . I f y o u w e re m o d e rn , y o u r s o ftw a re
in g l y h a rd t im e k e e p in g u p w it h M ic r o s o ft. w as in s ta lle d fro m a CD-ROM d riv e . T he s p e e d o f t h a t d riv e
m a tte r e d , a n d i f y o u w a n te d t o be a m o n g t h e c o o l k id s ,
T h e o t h e r p a rts o f t h e o p e r a tin g s y s te m w e re im p re s s iv e y o u n e e d e d a d r iv e t h a t w a s " 2 4 x o r fa s te r ." Y ou p ro b a b ly
t o o . USB w a s s u p p o r te d f o r t h e f i r s t tim e , w h ic h w a s a a ls o lo o k e d a t an SVGA CRT m o n it o r b a c k t h e n . T h a t's r ig h t

codem ag.com C O D E : 25 Y ears A g o 9

 F ig u re 2 : A ty p ic a l se tu p o f a PC ru n n in g W indow s 98, s p o rtin g b o th a 3 Vz-inch flo p p y
d riv e and a CD d rive

kids: We were lo o k in g a t 80 0 x60 0 pixels o f re s o lu tio n back fo r a w h ile , i t w a s n 't u n t il th e Late 90s t h a t I s ta rte d
th e n , and we loved i t ! Now g e t o f f m y law n! e x p e rim e n tin g w ith using a la p to p as m y m ain w o rkh orse
m achine . I s t i l l s w itc h e d back and fo rth be tw een t h a t
B u t W ind ow s 98 w a s n 't th e o n ly M ic ro s o ft o p e ra tin g sys­ approach and m ore c o n v e n tio n a l d es kto p se tup s (espe ­
te m t h a t was p o p u la r in th e Late 90s. M ic ro s o ft also c ia lly fo r g a m in g ) b u t h a v in g la p to p co m p uters w ith de­
c o n tin u e d to d e v e lo p W ind ow s NT as th e o p e ra tin g sys­ c e n t hors epo w er and rea son ab ly good c o lo r d is play s ju s t
te m o f c h o ic e fo r e n te rp ris e servers. NT4 was released h a d n 't been a th in g m uch e arlie r.
in 1996 and w o u ld be h e a v ily in use f o r a lo n g tim e to
com e. I t was b u ilt on NT 3 .5 and NT 3 .5 1 , w h ic h had i n ­ S peaking o f g a m in g : L oo king a t p o p u la r t itle s , th is was
tro d u c e d th e 3 2 - b it k e rn e l and a new f ile system (NTFS), w hen we s ta rte d re c o g n iz in g key players and brands t h a t
and A c tiv e D ire c to ry . I t 's a g re a t o p e ra tin g syste m , and s till have s ig n ific a n c e to d a y . Valve released CBA
H a lf - L if e
we s t i l l use th e successors o f m uch o f t h a t te c h n o lo g y (la te r, H a lf - L if e 2 was th e s ta r t o f Valve's now d o m in a n t
in m odern v e rs io n s o f W ind ow s to d a y . M any d ev e lo p ers S te a m g a m in g m arketpla ce) in 199 8. S ta r C r a ft e v e n tu a lly
even o p te d to use NT4 as th e o p e ra tin g system o f ch o ic e sparked c o m p e titiv e g a m in g and eS ports. U n r e a l by Epic
fo r t h e ir d e v e lo p m e n t m ac h in e s back th e n . I t was a b it Games was w here th e U n re a l E n g in e o rig in a lly came fro m .
c lu n k ie r fo r e nd -u se rs th a n W indow s 9 8 , b u t i t also had E n s e m b le S t u d io s (b a s ic a lly M ic ro s o ft) released A g e o f E m ­
a v e ry s o lid core. For te c h n ic a l p e o p le , i t w a s n 't a bad p ir e s , and B io w a r e gave us th e f ir s t B a ld u r 's G a te game
tr a d e - o ff a t a ll. (F ig u re 3 ) . T his m ig h t con fu se you because 2023's gam e
o f th e y e a r is also B a ld u r 's G a te , b u t th a t's th e successor
T his was also w hen Laptop c o m p uters becam e fa r m ore to th e classic gam e. B oth are h ig h ly recom m ended to any
p o p u la r and m ain stre a m . A lth o u g h th e y 'd been around a vid PC gam er.

10 C O D E : 25 Y ears A g o codem ag.com

Figure 3 : The is om e tric (and very low-res and fuzzy) goodness th a t is Baldur's Gate 1 in 1998.

B u t PC g a m in g w a s n 't a ll th e re was. Sony had alre ady I f you were tr u ly d arin g , you even dabbled in XML, w hich
released th e ir P la yS ta tio n brand a fe w years e arlie r, and was a new fangled standard based on angle-b rackets th a t
M ic ro s o ft was hard a t w o rk on th e f ir s t X box, w h ic h was to kinda looked lik e th e th e n -s till-n e w HTML, b u t i t had more
becom e a c o m p e tito r to P la yS ta tio n 2 as w e ll as N in te n ­ struc ture. To w h a t end, m any people w ondered? There was
do's GameCube, w h ic h it s e lf was a successor to N in te n do 's much sp ec u la tio n a b o u t web browsers being able to make
su cces sfu l N in te n d o 64 (F ig u r e 4 ) . Games lik e M o n o K a r t m ore sense o f XML th a n HTML, b u t m any d ou bted its po ­
were a ll th e rage on t h a t p la tfo rm b u t w o u ld u ltim a te ly te n tia l. I rem em ber s ittin g in a keynote a t a sizable com ­
be surpassed by th e am azing n e x t-g e n e ra tio n experiences p u te r conference in th e N etherlands, w here th e presenter
p ro vid e d by th e Xbox and th e PS2. B u t t h a t was s t i l l in
th e fu tu re .

Software Development
The w o rld o f s o ftw a re d e v e lo p m e n t in th e Late 90s was
b o th in flu x and also re la tiv e ly ca lm , in h in d s ig h t. I t
d id n 't seem Like i t a t th e tim e , because we had to com e to
g rip s w ith th e re a litie s o f th e in te r n e t and m any so ftw a re
develop ers were s t i l l w o n d e rin g w h e th e r th is g ra p h ic a l
user in te rfa c e experien ce was a c tu a lly good fo r business
a p p lic a tio n s and m any DOS d ie -h a rd s were s t i l l h o ld in g
on by th e skin o f th e ir te e th . B u t th e de ba te had p re tty
m uch been s e ttle d fo r m o s t o f us th e n : W indow s d es ktop
apps were w h a t serious develop ers were b u ild in g , and you
ju s t c o u ld n 't ig n o re th e in te r n e t e ith e r. We w ere d e v e lo p ­
in g w h a t seem ed to be re la tiv e ly s o p h is tic a te d in te ra c tiv e
w eb pages a t th e tim e th ro u g h th e use o f IS AP I w eb ­
server e xten sion s. J u s t im a g in e th e pow er: R ather th a n
w r itin g s ta tic file s w ith a .h tm l e x te n s io n , th e user co uld
h it a URL and a piece o f code co u ld respond, fig u re o u t
w h a t th e user w a n te d , and send back a s trin g o f HTML
t h a t was ge nerate d on th e fly ! I t seem ed as th o u g h w e'd
to u c h e d th e fu tu re (and so we h a d !). F ig u re 4 : The N in te n d o 64 was one h o t g a m in g device in th e m id -to -la te 90s!

codem ag.com C O D E : 25 Y ears A g o 11

 Figure 5: The Login screen to M icrosoft's ill-fa te d c o m p e tito r to th e e n tire in te rn e t was called
"The M icrosoft N etw o rk." The name survives as MSN, w hich is now one o f many web sites.

que stion ed th e p o in t o f XML and pre dicte d i t w o u ld go m any s e c o n d -tie r Languages t h a t ac hie ved a w ide degree
nowhere. He also pre dicte d m any o th e r th in g s , such as th e o f p o p u la rity , such as V is ua l FoxPro, D elp hi, Pearl, and
dem ise o f M icros oft. He que stion ed th e c o nc e pt o f "v is u a l" P yth o n . Some o f th es e rose from th e re , w h ile o th e rs were
de ve lo p m e n t (as in V isual Basic o r V isual C++). L ite ra lly e ith e r a t e n d -o f-L ife o r Lost s ig n ific a n c e . A good exam ple
none o f his p re d ic tio n s came tru e . I fo rg o t th e name o f th e o f th e la tte r is V is ua l Basic, w h ic h had its heyday in th e
presenter, b u t I o fte n w ondered w h a t became o f him . w o rld o f W indow s de sk to p a p p lic a tio n d e v e lo p m e n t. It's
cle a r t h a t its approach to s o ftw a re d e v e lo p m e n t has been
The m os t p o p u la r p ro g ra m m in g Languages o f th e Late am ong th e m os t im p a c tfu l c o n trib u tio n s in business ap ­
90s were V is u al Basic, C++, and Java. There were also p lic a tio n d e v e lo p m e n t, p e rio d . I t s t i l l c o n tin u e d to do so
fo r several decades and in to th e w o rld o f .NET, even when
C# to o k th e crow n o f b e in g th e m os t p o p u la r language in
th e M ic ro s o ft .NET ecosystem .

C# it s e lf was born as a b ra in c h ild o f A nders H ejlsberg in

th e Late 90s (and p u b lic ly an no un ced in th e sum m er o f
2 0 0 0 ). A nders is consid ered th e fa th e r o f m any pro gra m ­
m in g languages and has c le a rly had a d ra m a tic im p a c t
on p ro gra m m in g Languages in gen era l. He's th e o rig in a l
a u th o r o f Turbo Pascal and th e C hie f A rc h ite c t o f D elph i.
Once he jo in e d M ic ro s o ft as a T e ch n ica l Fellow, he created
C# ( o rig in a lly codenam ed Cool) in an e ffo r t to f ix some
o f th e s h o rtc o m in g s o f Java. He also la te r cre ate d th e
p o p u la r T y pe S c ript Language to f ix th e s h o rtc o m in g s o f
J a v a S c rip t. On a m ore pe rs on a l n o te , A nders is also one
o f th e m os t im p re ssiv e y e t n ic e s t pe op le I ever had th e
pleasure o f w o rk in g w ith ( f u ll d is clo sure : I w orked fo r th e
M ic ro s o ft V is ua l S tu d io te am as a c o n tra c to r in th e la te
90s, b u t m y pe rs on a l in v o lv e m e n t w ith C# was n o t hu ge).

A lth o u g h M ic ro s o ft had been v e ry s tro n g in W indow s de­

v e lo p m e n t to o ls , w ith th e m ain w orkhorses b e in g V isua l
Basic, V is ua l C++, and V is u al FoxPro, M ic ro s o ft had also
o rig in a lly m issed th e b o a t on th e in te r n e t wave, t h in k ­
in g its ow n c o m p e tin g M ic ro s o ft N e tw ork (MSN) had a
re al chance o f c o m p e tin g w ith th e in te r n e t (and also
Com puServe) as a w a lle d garden w ith in th e W indow s eco ­
F ig u re 6 : The M ic ro s o ft V is ua l In te rD e v CD sleeve system . You can be fo rg iv e n fo r n o t rem em be rin g MSN

12 C O D E : 25 Y ears A g o codem ag.com

as a separate te c h n o lo g y . The te c h n o lo g y is , in fa c t, so
SPONSORED SIDEBAR
obscure, t h a t I had a hard tim e fin d in g any screen sho ts,
and I had to s e ttle fo r th e lo g in screen (F ig u re 5 ) . In CODE Is Hiring!
an a tte m p t to ca tch up, M ic ro s o ft released A c tiv e Server
Pages (ASP) as a te c h n o lo g y , and V is u al In te rD e v as a CODE Staffing is accepting
d e v e lo p m e n t e n v iro n m e n t fo r i t . I t was a f ir s t a tte m p t to resumes for various
create an IDE fo r w eb d e v e lo p m e n t in s p ire d by th e ideas open positions ranging
e s ta b lis h e d by V is u al Basic. The IDE was also shared w ith from junior to senior
V is u al J+ + , w h ic h was M ic ro soft's ve rs io n o f a Java d e v e l­ roles. We have multiple
o p m e n t e n v iro n m e n t. openings and will consider
candidates who seek
A lth o u g h n e ith e r V is u al In te rD e v n o r V is ua l J+ + were full-time employment or
contracting opportunities.
aro und lo n g (and th e lo go was one o f th e b e tte r pa rts
For more information, visit
o f th e w h o le V is u al In te rD e v s u ite , as seen in F ig u re
www.codestaffing.com.
6 ) , th es e tw o p ro du cts s t i l l la id th e fo u n d a tio n fo r w h a t
w o u ld u ltim a te ly becom e one o f th e m o s t su cces sfu l s o ft­
ware d e v e lo p m e n t IDEs and e n v iro n m e n ts ever: V is ua l
I t looks lice y o u 're
S tu d io fo r W indow s. V is u al S tu d io was th e f ir s t IDE t h a t
was Language- and te c h n o lo g y -a g n o s tic and a llo w e d de­ w r itin g a le tte r.
ve lo pe rs to sta y in th e same IDE, regardless o f w h e th e r
th e y w a n te d to develop in V is ua l Basic, C++, C#, o r q u ite W o u ld y o u lice h e lp ?
a fe w o th e r languages, and regardless o f w h e th e r th e y
c G e t h e lp w ith
were b u ild in g fo r W indow s, th e Web, o r m any o th e r p la t­
w r itin g th e le tte r
form s o f th e fu tu re . However, as w e're Looking 25 years
back to th e la te 90s, none o f th is had happened y e t, c Jus t ty p e th e
and V is ua l In te rD e v and th e f ir s t ve rs io n o f A c tiv e Server le tte r w ith o u t
Pages is a ll we g o t. I d o n 't m iss th o s e days. h e lp

r D o n 't s h o w m e
T h a t sums up w here we were 25 years ago fro m a so ftw a re
th is tip a g a in
deve loper's p o in t-o f-v ie w . We were on th e cusp o f a Lot o f
changes, b u t we h a rd ly rea liz ed i t back th e n . N everthe ­
less, a lo t o f th e th in g s we w ill lo o k a t in th e n e x t a rtic le ,
w hen we lo o k back 20 years, was th e in e sca pa ble fu tu re
based on w h a t was b u ilt in th e la te 90s. A nd Let's n o t
fo rg e t t h a t th e Late 90s is also th e b ir th o f C lippy, th e n o t
so belov ed g ra n d fa th e r o f th e p o p u la r ChatGPT.

M arkus Egger

COD YEARS

codem ag.com C O D E : 25 Y ears A g o 13

O N LIN E Q U IC K ID 2403031

Passkey Authentication
In a previous article in CODE Magazine, https://www.codemag.com/Article/2209021/FID02-and-WebAuthn, I talked about
FID02 and WebAuthn. Look, passwords suck, but finally, as an industry, we're getting behind getting rid of them for real. The
FIDO2 standard bakes in enough capabilities to completely eliminate passwords, and passkeys offer the convenience to be a

g re a t passw ord re p la ce m e n t. Yes, passw ord s te a lin g is tio n . However, I'v e ta ke n th e s h o rtc u t fo r tw o reasons.
g o in g to be h is to ry and a lo n g w ith i t , th in g s such as The f ir s t reason is b re v ity o f code. The second reason
p h is h in g . This s a fe ty w ill g ra d u a lly be so c o m p le te ly b u ilt is t h a t a n y tim e I s ta r t d e b u g g in g , m y a p p lic a tio n resets
across th e in te r n e t t h a t o u r g ra nd m oth e rs w ill fe e l co m ­ back to zero. T his res et m eans t h a t be tw een runs o f th e
fo rta b le us ing i t . O f course, s e c u rity is a gam e o f c a t and a p p lic a tio n , I m u s t d e le te th e passkey t h a t th e user re g is ­
m ouse. The th re a ts w ill ju s t m ove elsew here, lik e h ac king te re d . T his is an okay w orkarou nd fo r dem o code. A lso , th e
in to ba ck -e nd system s, s o c ia l e n g in e e rin g , etc. hard -co de d user I ' l l be using is " s a h il@ lo c a lh o s t.c o m ".

T h a t is n o t w h a t th is a rtic le is a b o u t. This a rtic le is a b o u t A lth o u g h th e FID02 p ro to c o l and th e W ebA uthn stand ard
Sahil. Malik im p le m e n tin g passkey a u th e n tic a tio n in y o u r w eb a p p li­ are n o t Language s p e c ific , th e bro w se r code m us t be w r it ­
www.winsm arts.com c a tio n s . I ended th e pre vio us a rtic le by sh o w in g how to te n in J a v a S c rip t. The server code can be im p le m e n te d
@ sahilmaLik use hardw are-based FID02 keys, such as YubiKeys to au ­ in any Language y o u w is h . There are lib ra rie s fo r va rio us
th e n tic a te w ith M ic ro s o ft Entra ID (fo rm e rly Azure A D ). languages t h a t y o u can use. For m y purposes, I ' l l be us­
S ahil M alik is a M icrosoft
A lth o u g h th a t's a p e rfe c tly capable o p tio n , w h a t i f yo u're in g NodeJS.
MVP, INETA speaker,
n o t using Entra ID? Or w h a t i f y o u t r u ly w ish to un d er­
a .NET author, consu ltant,
sta nd w h a t's g o in g on b e h in d th e scenes? I w o n 't be fo c u s in g on NodeJS basics here.
and trainer.

S ahil loves in te ra c tin g In th is a rtic le , I'm g o in g to b u ild an e n d -to -e n d a p p lic a ­ W ith th es e a ss um ptio n s, le t's g e t s ta rte d .
w ith fe llo w geeks in real tio n t h a t shows how to im p le m e n t FID02 a u th e n tic a tio n
tim e. His talks and tra in ­ b o th on th e c lie n t side and server side. I'Ll use t h a t as an
ings are f u ll o f hum or and excuse to e xp la in th e va rio u s d e ta ils in th e spec, and I ' l l
Project Set Up
pra ctical nuggets. use i t to d e m o n s tra te passkeys. The basic p ro je c t s tru c tu re I'm using can be seen in jihgfedcbaZYX
F ig ­
u re 1 .
His areas o f expertise are
cross-platform M obile app A ssu m ptio n s T his Looks lik e a sim p le NodeJS p ro je c t. T his a p p lic a tio n
developm ent, M icrosoft
The FID02 sta nd ard is q u ite fle x ib le . I t considers va rio us is b u ilt us ing express and takes a dep endency on th e f o l­
a ny th in g , and security
nuances, such as th e q u a lity o f keys y o u w ish to c o n tro l, lo w in g npm packages.
and id e n tity .
how m any c re d e n tia ls a user can create fo r a c e rta in URL,
w h a t kin ds o f passkeys are s u p p o rte d , e tc . A lth o u g h I The p ro je c t uses d o te n v to ho ld e n v iro n m e n t v a ria ble s
w o n 't have th e o p p o r tu n ity to d em o n stra te every s in g le e q u iv a le n t in th e .e n v file . D o te nv and .e n v file s are a
p o s s ib ility in th is a rtic le , fo r b re v ity and th e sake o f ha v­ g re a t w ay o f m a na ging c o n fig u ra b le in p u ts to y o u r p ro ­
in g a c o m p le te a p p lic a tio n w o rk in g by th e end o f th is gram . The th o u g h t is t h a t y o u can s p e c ify values as a .e nv
a rtic le , le t's s ta rt w ith some ass um ptio ns . f ile in y o u r lo c a l dev e n v iro n m e n t. When th e .e n v file is
m is s in g , i t can p ic k th o s e same values fro m an e n v iro n ­
T his a rtic le is g o in g to show dem o code. This code is n o t m e n t v a ria b le . T his is in c re d ib ly u s e fu l w hen y o u package
in te n d e d fo r p ro d u c tio n use. There are s e c u rity s h o rtc u ts y o u r a p p lic a tio n and s h ip i t as, say, a Docker c o n ta in e r.
in th e code Tm a b o u t to de m on stra te . T y pically, y o u 'd create a .e n v as a sam ple, check i t in ,
th e n add i t to .g itig n o re and a llo w develop ers to add sen ­
A lth o u g h th e FID02 sta nd ard a llo w s a user to create m u l­ s itiv e in fo rm a tio n . Or y o u can create a .env.exa m ple in
tip le c re d e n tia ls fo r th e same w e b s ite , I'm g o in g to re­ th e ro o t o f y o u r p ro je c t. T his is a g re a t w ay to reduce th e
s tr ic t m y a p p lic a tio n to using a s in g le c re d e n tia l fo r th e im pe da nc e m ism atch betw e en p ro d u c tio n and dev.
give n w e b s ite . I'Ll ta lk th ro u g h w h a t m u ltip le c re d e n tia ls
m eans fo r a user and th e fa c ilitie s W ebA uthn a llo w s t h a t I'm also ta k in g a dep endency on express and express­
help a user p ic k a c e rta in c re d e n tia l o r a llo w s y o u r a p p li­ session. These are ex pres s-re la te d packages t h a t a llo w me
c a tio n to s p e c ify w h ic h e x p lic it c re d e n tia l is a ccep ta ble to create a basic w e b s ite t h a t s u p p o rts a sim p le U I, and
fo r a g iv e n purpose. som e ba ck-end code t h a t s u p p o rts an A P I. A lso , i t a llo w s
me to s u p p o rt sessions.
FID02 a llo w s y o u to create c re d e n tia ls fo r a g iv e n URL
as lo n g as y o u r w e b s ite runs on HTTPS. For d e v e lo p m e n t F ina lly, I'm ta k in g a dep endency on " f id o 2 - lib " npm pack­
purposes, y o u can also use lo c a lh o s t w ith o u t HTTPS, and age. A lth o u g h I co u ld w rite th e FID02 code m ys elf, I re­
th a t's w h a t I ' l l be u sin g. a lly d o n 't w ish to re in v e n t th e w h e e l and spend m y w e ek ­
end im p le m e n tin g th e w e ll-d o c u m e n te d 1 9 -s te p v a lid a ­
To keep th in g s sim p le , I'm g o in g to use session s ta te as tio n lo g ic w hen som eone else has alre ady done i t and it's
m y database. A g a in , th is is a huge s h o rtc u t I'm ta k in g , been peer review ed and w e ll te s te d .
and th is is , by no means, a p ro d u c tio n -re a d y a p p lic a tio n .
In re a lity , y o u 'd use a p e rs is te n t storag e, such as a da­ The "d e p e n d e n c ie s " s e c tio n o f m y package.json Looks lik e
tabase, to m a in ta in user re g is tra tio n and lo g -in in fo rm a ­ th is :

Passkey Authentication codem ag.com

 "dependencies": {
"dotenv": " A 16.3.1",
"express": " A 4.18.2",
"express-session": " A 1.17.3",
"fido2-lib": " A 3.4.3" d tS e a r c K
The re s t o f m y package.json is p re tty p la in v a n illa .

In th e re s t o f m y p ro je c t, th e .e n v f ile holds m y e n v iro n ­

Instantly Search
m e n t va ria b le s , w h ic h can be seen here:

RPDisplayName = "localhost"
Terabytes
RPID = "localhost"
RPOrigin = "h ttp ://lo c a lh o s t"
RPPort = "80"

These are a ll values I need e ith e r fo r th e FID02 stand ard

o r to run m y s ite . For th e p o rt I w ish to run o n, i t w o u ld dtSearch's docum ent filters support:
be nice i f th is were c o n fig u ra b le . Once referenced in my
• popular file types
.env, I can use th es e values:
• emails with multilevel attachments
const env = dotenv.configO.parsed;
console.log(env.RPID);
• a wide variety of databases
• web data
The re s t o f m y code co ns is ts o f som e serv er-s id e and some
c lie n t-s id e code. The server-side code is re sp on sible fo r
se rv in g th e c lie n t-s id e code as a sim p le w e b s ite and ex­
po sin g c e rta in APIs t h a t are re qu ired fo r th e FID02 re g is ­
tr a tio n and a u th e n tic a tio n to succeed.
Over 25 search options including:
XWVUTSRQPONMLKJIHGFEDCBA • efficient multithreaded search
The c lie n t-s id e code presents a ve ry s im p le user in te rfa c e
w here I show th e user a t e x t box p ro m p tin g th e user to
• easy multicolor hit-highlighting
e n te r th e ir usernam e, and c lic k th e re g is te r o r lo g in b u t­ • forensics options like credit card search
to n as needed. A d d itio n a lly , fo r debug purposes, I can
show th e user a s ta tu s message sh o w in g th e o u tp u t o f
th e ir m o s t re c e n t a c tio n . The user in te rfa c e in a c tio n can
be seen in F ig u re 2 . 1 hope y o u 're im pressed by m y design
s k ills . Developers:
The server-side code also exposes fo u r POST m eth o ds : be-
• SDKs for Windows, Linux, macOS
g in R e g is tra tio n , e n d R e g is tra tio n , b e g in L o g in , and end- • Cross-platform APIs cover C++, Java
L o g in . I ' l l e x p la in each o f th es e in d e p th s h o rtly . They're
and current.NET
im p le m e n te d in th e / lib s / a u t h n . js file . A p a r tia l s n ip p e t
o f s e ttin g up one o f th es e rou te s can be seen below . • FAQs on faceted search, granular data
classification, Azure, AWS and more
import express from 'express';

ro u te r. use(express. j son( ) ) ;
ro u te r.p o st( ' /beginR egistration' ,
async (req, res) => {

})
Visit dtSearch.com for
• hundreds of reviews and case studies
export default router;
• fully-functional enterprise and
The a u th n .js f ile also makes use o f sessions t h a t are s e t developer evaluations
up as below . O f n o te , th is is c o m p le te ly in sec ure code.
You'd w a n t to use secure sessions in p ro d u c tio n , and you
w o u ld n 't w a n t to save user in fo rm a tio n in sessions a t a ll. The Smart Choice for Text
B u t th is is dem o code.
Retrieval® since 1991
import session from 'express-session';
d tS e a rc h .c o m 1-800-IT-FIND S
router.use(sessi on({
secret: 'keyboard c a t ',

codem ag.com Passkey Authentication 15

 resave: false, are lib ra rie s and th e user in te rfa c e is super sim p le . These
saveU ninitialized: true file s are served w ith o u t a u th e n tic a tio n using th e th re e
})) jihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA line s un d er th e s e t-u p m id dlew are c o m m e n t in L is tin g 1 .

By s e ttin g up th is code, I can now s e t a session v a ria b le , F ina lly, le t s ta lk a b o u t /L ib s /u s e r.js . T his is m y database
as below . o f e x a c tly one user. I to ld y o u th is was dem o code. I d id n 't
w a n t to b o th e r s e ttin g up a database, so I hard -co de d a
req.session.challenge = response.challenge; user here. The code fo r users.js can be seen in L is tin g 3 .
req.session.save(); The th o u g h t here is t h a t I'm w irin g an a p p lic a tio n w here
th e user's lis t is alrea dy kn ow n. Let's say an ad m in has
A nd th is v a ria b le can th e n be read a t a la te r tim e , lik e so: s e t up th e users, and th e user is ex pe cted to pe rfo rm reg ­
is tra tio n fo llo w e d by log in . D uring re g is tra tio n , th e user
req.session.challenge co u ld pro vide a sec ret, e tc . b u t I'm n o t g o in g to b o th e r
w ith t h a t fo r dem o code. The users.js f ile creates a JSON
Once th es e ro utes and Login are s e t up in a u th n .js , I can o b je c t t h a t acts as m y po o r-m an 's database. Now, g iv e n a
s im p ly im p o r t th e m in server.js, w h ic h serves as th e m ain usernam e, I can s im p ly Look up a user in serv er-s id e code,
e n try p o in t fo r m y a p p lic a tio n . The f u ll code fo r server.js as belo w , and w o rk w ith th e user.
can be seen in L is tin g 1 . L e ts un de rs ta nd i t b it by b it .
const user = users[req.body.username]
A t th e very to p , I'm im p o rtin g Express, w hich I in te n d to
use to s e t up m y basic web app, fo llo w e d by dote nv, w hich I can even m o d ify th e user by a d d in g to th e C redentials
helps me w o rk w ith c o n fig u ra tio n in fo rm a tio n . I th e n im ­ p ro p e rty , and w hen I s to p d e b u g g in g , m y user o b je c t re­
p o rt a u th n .js , w hich holds my server-side lo g ic fo r APIs. sets back to its o rig in a l s ta te .

I th e n s e t up m id dlew are to serve s ta tic file s . My ap ­ The s tru c tu re o f th e user o b je c t is d rive n by th e W ebAuthn
p lic a tio n , as can be seen in F ig u re 1 , co ns is ts o f an in - stan da rd .
d e x .h tm l w ith som e v e ry sim p le user in te rfa c e , and th re e
F ig u re 1 : My p ro je c t J a v a S c rip t file s . You can see th e f u ll code fo r m y in d e x , The " id " provid es th e user ha nd le o f th e user a c c o u n t. A
s tru c tu re h tm l file in L is tin g 2 . As can be seen, tw o o f th o s e file s user ha nd le is an opaque b y te sequence w ith a m axim um
size o f 64 b ytes , and is n 't m ea nt to be d is p la ye d to th e
user. This ID can be considered u n iq u e and de cision s are
L is tin g 1 : The server.js m ain e n try p o in t fo r m y a p p lic a tio n to be made based on th is ID , n o t based on d is p la y name

import express from 'express'; o r name. It's recom m ended to use a c o m p le te ly random
import dotenv from 'dotenv'; va lu e o f 64 by tes fo r th is ID . I have ha rd -co ded th is to
import authn from ' . /lib s /a u th n . j s ' ; 12 34 as a s h o rtc u t.

const app = express();

The nam e and displayN am e are p ro p e rtie s o f th e user ac­
/ / load env c o u n t d u rin g re g is tra tio n , w h ic h is a h u m a n -p a la ta b le
const router = express.RouterQ; nam e fo r th e user a c c o u n t, in te n d e d o n ly fo r dis play .
const env = dotenv.configO .parsed; In m y a p p lic a tio n , I'v e s e t th is to "S a h il M alik". You can
im a g in e t h a t in s e lf-s e rv ic e sign ups, th e user can s p e c ify
/ / setup middleware
app.use(express. s t a t ic (' / ' )) th e ir own name.
app.use(express. j son( )) ;
app.use(express. s ta ti c ( ' . / p u b lic ') ) ; The c re d e n tia ls p ro p e rty is in te r e s tin g . I t s to re s th e List
o f c re d e n tia l o b je c ts fo r th e user. For in s ta n c e , i f th e
a p p .g e t(7 ', function (req, res) {
res.sendF ile(' in d e x .h tm l'); user re g is te rs an iC lo ud passkey, a Y u biK e y c re d e n tia l,
} ); a Chrome p ro file passkey, and a BLE and NFC passkey
u s in g t h e ir Sam sung p ho ne , th e user w ill now have fiv e
app.use(7authn', authn);
c re d e n tia ls fo r th e same RP (re ly in g p a rty ). For m y a p ­

app.listen(env. RPPort, () => { p lic a tio n , I'm g o in g to r e s tr ic t th e user to o n ly one cre ­

console.lo g ('Your app is lis te n in g on port ' d e n tia l. I f y o u d id have m u ltip le c re d e n tia ls , th e serve r
+ env.RPPort); can s p e c ify w h ic h c re d e n tia ls are a llo w e d o r d is a llo w e d .
} ); A d d itio n a lly , th e W eb A uthn s ta n d a rd has th e a b ilit y to
a u to m a tic a lly a llo w th e user to p ic k th e la s t used key, o r
to p ro m p t th e user to p ic k fro m a List o f keys, and l i m i t ­
s a hil@ lo ca lh os t.co m R e g is te r Login in g t h a t lis t by e x c lu d in g c e rta in c re d e n tia ls , as needed.

You can im a g in e t h a t th is w o u ld be u s e fu l w here a cer­

" c r e d id " : " w T R h X y F Y E o 3 w l5 0 x p g - X Y Z S y Y D g ' ta in a c tio n can be a llo w e d w ith an iC loud passkey. B u t
" p u b lic K e y ” : " --------- B E G IN P U B L IC K E Y ------
fo r c e rta in e le v a te d a c tio n s , y o u re q uire a Y ubiK ey w ith
" a a g u i d " : ” - _ w w B x V 0 T s y M C 2 4 C B V fX v Q " ,
a tte s ta tio n , e tc . A tte s ta tio n is b u ilt in to th e FIDO and
" p re v C o u n te r" : 0 ,
W ebA uthn p ro to c o ls , w h ic h enables each re ly in g p a rty to
" fla g s ” : {},
use a c ry p to g ra p h ic a lly v e rifie d ch ain o f tr u s t fro m th e
" ty p e ” : " p u b lic -k e y "
device's m a n u fa c tu re r to choose w h ic h s e c u rity keys to
tr u s t, o r to be m ore s k e p tic a l o f, based on th e ir in d iv id u a l
F ig u re 2 : The a p p lic a tio n user in te rfa c e in a c tio n needs and concerns.

16 Passkey Authentication codem ag.com

In th e ir c u rre n t fo rm , passkeys o ffe r no a tte s ta tio n . In en ­ L is tin g 2 : A p p s e ttin g s .js o n
te rp rise scenarios, a tte s ta tio n can be custom ized and con ­
< !DOCTYPE html>
tro lle d as I blogged a b o u t here: h ttp s ://w in s m a rts .c o m /
chtml lang="en">
p as ske ys-a n d-en te rp ris e-au th en tic atio n-75 0e e6 33 2c 25 . A l­
te rn a tiv e ly , you can also have pre-registered hardware keys, <head></head>
so when I m a il you a key, you m ust reg is te r th e sp ec ific key
I m ailed yo u, n o t ju s t any key you b o u g h t from a store. This <body>
can be u se fu l i f you w ish to have grea te r confiden ce in th e <input type="text" id="username"
value="[email protected]" />
hardware keys being used in y o u r ente rprise , fo r instan ce, i f
cbutton id="registerB utton">Register</button>
you d o n 't w a n t keys w ith user fla sh ab le ROM because th a t
cbutton id="loginB utton">Login</button>
kind o f defeats th e purpose. Or you d o n 't w a n t softw are
keys because th e y can be easily shared across th e in te rn e t. <br/>
Locking a ll th a t down is w h a t a tte s ta tio n is designed to do. <pre><div i d="userMessage"x/div></pre>

<script src="/js/base64U rl. j s"></script>

A n o th e r w ay to Look a t th is is : I f y o u w a n t g re a te r secu­
<script s rc = "/js /jq u e ry .js"> < /script>
r it y a t th e ris k o f usage, y o u 'll re qu ire t ig h te r and m ore
<script s r c = " /js /c lie n t. j s"></script>
c o n tro lle d a tte s ta tio n . B u t i f y o u w ish to have consum er </body>
scenarios and o ffe r th e le a s t fr ic tio n and w id e s t audience,
y o u 'll re qu ire no a tte s ta tio n . Even w ith no a tte s ta tio n , </html>
y o u 're s t i l l m ore secure th a n th e n ic e s t, m os t co m p le x
passw ord y o u can use. So it's s t i l l a w in o ve r w h a t we
do c u rre n tly . T his is w hy passkeys are som etim e s calle d L is tin g 3 : users.js
passw ord re p la ce m e n t.
le t users = {
"[email protected]" : {
For th e purposes o f th is a rtic le , th o u g h , I'LL re qu ire no a t­ id : "1234",
te s ta tio n . I hope to ta lk m ore a b o u t a tte s ta tio n in fu tu re name : "Sahil M alik",
a rtic le s . displayName: "Sahil M alik",
credentials: []

Before I go m uch fu rth e r, le t's ta lk a b it a b o u t th e "c re ­ }

}
d e n tia l" o b je c t th o u g h . The c re d e n tia l o b je c t is also ex­
p ec te d to be in a c e rta in fo rm a t. A user is associa ted export default users
w ith an array o f c re d e n tia ls , w h ic h I w ill Lim it to one, fo r
s im p lic ity . The c re d e n tia l o b je c t c o n ta in s th e fo llo w in g
p ro p e rtie s : a u n iq u e ID id e n tifie r, an a tte s ta tio n ty p e , id e n tifie r s to re d as an array o f th e a u th e n tic a to r m odel
th e ty p e o f tra n s p o rt used, c re d e n tia l fla g s , and re la te d b e in g s o u g h t.
a u th e n tic a to r in fo rm a tio n .
The a u th e n tic a to r in fo rm a tio n also in c lu d e s sig n C o u n t
The c re d e n tia l o b je c t has an ID t h a t u n iq u e ly id e n tifie s (th e s ig n a tu re c o u n te r v a lu e ) and clone w a rn in g s , w hich
th e c re d e n tia l. I f th e RP in s is ts on a c e rta in o p e ra tio n help th e RP d e te c t clo ne d keys. W ith each new lo g -in
re q u irin g a c re d e n tia l o f a c e rta in q u a lity , i t can pass o p e ra tio n , th e RP com pares th e sto re d s ig n C o u n t va lu e
th es e ID s as an array in th e a llo w e d C r e d e n tia ls p ro p e rty w ith th e new s ig n C o u n t va lu e re tu rn e d in th e a ss ertio n 's
to th e c lie n t. a u th e n tic a to r d a ta . I f th is new s ig n C o u n t va lu e is less
th a n o r e q u a l to th e sto re d value , a clo ne d a u th e n tic a to r
Each c re d e n tia l also has an a tte s ta tio n T y p e t h a t id e n ti­ m ay e x is t, o r th e a u th e n tic a to r m ay be m a lfu n c tio n in g .
fie s th e a tte s ta tio n fo rm a t used by a c e rta in a u th e n tic a ­ The clone w a rn in g is a s ig n a l t h a t th e a u th e n tic a to r may
t o r w hen th e c re d e n tia l was cre ate d. be c lo n e d ; in o th e r w ords, a t Least tw o copies o f th e cre­
d e n tia l p riv a te key m ay e x is t and are b e in g used in p ar­
The c re d e n tia l also c o n ta in s th e tra n s p o rt used, w h ic h a lle l. RPs sh o u ld in c o rp o ra te th is in fo rm a tio n in to th e ir
can be USB (a USB ke y), NFC (n e a r fie ld c o m m u n ic a tio n ), ris k s c o rin g . W h ethe r th e RP updates th e s to re d sig n C o u n t
BLE (b lu e to o th Low e n e rg y ), h y b rid (a m ix tu re o f one o r va lu e in th is case o r n o t, o r fa ils th e a u th e n tic a tio n cer­
m ore tra n s p o rts ), o r in te r n a l (an a u th e n tic a to r t h a t ca n ­ em ony o r n o t, is R P -specific.
n o t be rem oved from th e d e v ic e ).
The a u th e n tic a to r also co nta in s an A u th e n tic a to r A tta c h -
The c re d e n tia l c o n ta in s c re d e n tia l fla g s , w h ic h help you m e n t p ro p e rty th a t allow s th e RP to express w hich au­
id e n tify w h e th e r th e user was p re sen t o r v e rifie d d u rin g th e n tic a to rs th e RP prefers, and th e c lie n t o n ly shows a
a u th e n tic a tio n and re g is tra tio n . Presence m eans t h a t th e preferred lis t o f a u th e n tic a to rs to th e user. For in stan ce , I
user has to to u c h a key, and v e rific a tio n m eans t h a t th e may be okay w ith using iCloud b u t n o t Chrome p ro file s, etc.
user has to to u c h a key and prove w ho th e y are v ia a PIN
o r b io m e tric s . A d d itio n a lly , a key can be b ac k-u p e lig ib le , W ith a ll th is ba ck grou nd o f m y p ro je c t s tru c tu re , now le t's
w h ic h d e te rm in e s w h e th e r a key can synch betw e en de­ foc us on b u ild in g th e a p p lic a tio n .
vices, such as iC loud passkeys.

F inally, th e re 's a u th e n tic a to r-re la te d in fo rm a tio n fo r each

Overall Application Logic
c re d e n tia l, w h ic h co ns is ts o f several ite m s . A u th e n tic a ­ My a p p lic a tio n consists o f tw o steps. The f ir s t is re g is tra ­
to r-re la te d in fo rm a tio n in c lu d e s an AAGUID (a u th e n tic a ­ tio n , th e second is log in . D uring re g is tra tio n , th e c lie n t­
t o r a tte s ta tio n g lo b a l u n iq u e id e n tifie r ) th a t's th e u n iq u e side Jav aS cript is expected to c a ll th e b e g in R e g is tra tio n

codem ag.com Passkey Authentication 17

 A P I, w hich generates a challeng e on th e server side along a tte s ta tio n expectations based on th e stored credential and
w ith o th e r preferences t h a t th e RP requires fo r an a ccep t­ generated challenge, and i f i t passes, th e user is Logged in .
able cre d e n tia l. The c lie n t th e n calls th e jihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA
n a v ig a to r.c re d e n -
tia ls .c r e a te A P I to generate a c re d e n tia l th a t's passed to T his is , in sum m ary, how FID02 and W ebA uthn w ork. Let's
th e server in th e e n d R e g is tra tio n ca ll. The e n d R e g is tra ­ see th is in a c tio n .
t io n c a ll va lid a te s i f th e generated c re d e n tia l is acceptable,
and i f i t is, i t saves th e c re d e n tia l on th e user o b je c t and
in fo rm s th e user o f a successful re g is tra tio n .
Registration
F irst, Let's foc us on re g is tra tio n s ta rtin g w ith th e c lie n t
Once th e user is registered, th e user can a tte m p t to Log in . side. Loo king a t L is tin g 2 , w hen th e user e nte rs th e ir
Logging in , again, has tw o steps. The fir s t is a c a ll to th e userID and presses th e R e g is te r b u tto n , y o u c a ll th e
b eg in Login API, w hich generates a challenge on th e server server-side b e g in R e g is tr a tio n m e th o d , as show n below .
and passes acceptable a u th e n tic a to r details to th e clie n t.
The c lie n t th en uses th e n a v ig a to r.c re d en tia ls .g e t m ethod var credOptionsRequest = {
according to th e return value o f lo g -in expectations from b e - username: $("#username").va l()
g in L o g in , and sends back th e p ro of o f a u th e n tic a tio n to th e };
server in th e en d L o g in API call. This is th en ve rifie d against
var credentialCreationOptions =
await _ fe tc h ( 1/authn/beginR egistration 1 ,
L is tin g 4 : B egin R e g is tra tio n re qu es t
credOptionsRequest);

"rp ": { The _ fe tc h m eth o d is a s im p le h e lp e r m eth o d I'v e w ritte n

"name": "localhost", to en ca ps u la te a POST c a ll.
"id " : "localhost"
}.
"user": { The idea is th a t th e c lie n t has expressed in te re s t in regis­
"id " : "1234", te rin g th e u s e rn a m e id e n tifie d and is asking th e server fo r
"displayName": "Sahil M alik", in fo rm a tio n on w h a t kind o f c re d e n tia l is acceptable. The
"name": "Sahil Malik" server th e n uses th e fid o 2 -lib npm package to generate
}. a response. W ith in th e response is c o n ta in e d a challenge
"challenge":
and o th e r d e ta ils , such as th e displayN am e th a t's show n to
" i hOh6rCD92T0xZNAi11PGG71xJswOqTKSNo11Jj G32s" ,
"pubKeyCredParams": [ th e user. I ' l l o m it th e Node.js code because you can eas­

{ ily fig u re t h a t o u t using th e d o c u m e n ta tio n o f th e fid o 2 -

"type": "public-key", Lib p ro je c t. The JSON o b je c t o f in te re s t th a t kic ksta rts th e
"a lg ": -7 a u th e n tic a tio n can be seen in L is tin g 4 . There are a lo t
}. o f in te re s tin g d e ta ils here. The " r p " id e n tifie s th e re ly in g
{ p a rty th a t th e brow ser w ill ensure m atches w here th e s ite
"type": "public-key",
"a lg ": -257 is ru n n in g . This p re tty much e lim in a te s p h is h in g . The user

} p ro p e rty co nta in s in fo rm a tio n a b o u t th e user, w h ich is

], shown to th e user d urin g re g is tra tio n . This can be seen as
"tim eout": 1800000, th e usernam e "S a h il M alik ", as show n in F ig u re 3 .
"a tte s ta tio n ": "none",
"authenticatorS election": {
The ch a lle n g e is a u n iq u e s trin g ge ne ra ted on th e server
"authenticatorAttachment": "p la tfo rm ",
" requireResidentKey" : false, side t h a t th e c lie n t m us t use in g e n e ra tin g th e c re d e n tia l.
"userverification ": "preferred" In th e e n d R e g is tra tio n le g , I ' l l v e rify t h a t th e same c h a l­
}, lenge was used as was ge ne ra ted on th e server. The tim e ­
"excludeCredentials": [] o u t m e n tio n s th e m illis e c o n d s w ith in w h ic h th e re g is tra ­
tio n m us t be c o m p le te d . I t requires no a tte s ta tio n , and

L is tin g 5 : End R e g is tra tio n

const user = users[req.body.username] await f2 1 . a tte statio nR esu lt(

const clientAttestationResponse = { response: {} }; clientAttestationResponse, attestationExpectations)
c li entAttestat i onResponse. rawld =
coerceToArrayBuffer(req.body.rawld, "ra w ld "); const credential = {
clientAttestationResponse.response.cli entDataJ SON = credld: coerceToBase64Url(
req. body. response.c li entDataJ SON; regResult.authnrData. ge t( "cred ld"), 'c r e d ld ') ,
clientAttestationResponse.response.attestationobject = publicKey:
coerceToArrayBuffer(req. body. response. attestationObj e c t, regResult.authnrData.get( "credentialPubli cKeyPem"),
"a tte s ta tio n o b je c t"); aaguid: coerceToBase64Url(
regResult.authnrData.get( "aaguid"), 'a a g u id '),
const attestationExpectations = { prevCounter: regResult.authnrData.get("counter")
challenge: req.session.challenge, flags: regResult.authnrData.get( "flags"),
o rig in : env.RPOrigin, type: 'public-key'
fa cto r: "e ith e r" };

user.credentials.push(credential)
const regResult =

18 Passkey Authentication codem ag.com

I'v e s p e c ifie d c e rta in c h a ra c te ris tic s o f th e a u th e n tic a to r pe rfo rm s T ouchlD (o r FacelD i f y o u are on iOS o r W indow s
I'm okay w ith . S p e c ific a lly , I'm sa ying t h a t I w a n t p la t­ H ello on W indow s, e tc .), and th is generates a c re d e n tia l.
fo rm a u th e n tic a to r. Because I'm on a Mac, I'm p ro m p te d
to use T ou ch lD , as can be seen in F ig u re 3 . Now th e user T his c re d e n tia l is s e n t to th e server in th e /e n d R e g is tra ­
tio n c a ll and can be seen below .

{
"username": "[email protected]",
" id " : "wTRhXyFYEo3wl50xpg-XYZSyYDg",
"rawld": "wTRhXyFYEo3wl50xpg-XYZSyYDg",
"type": "public-key",
"response": {
"a tte sta tio n o b je ct":
"clientDataJSON":
}
}

The a tte s ta tio n o b je c t and clientD ataJS O N p ro p e rtie s help

th e server v e rify i f th is re g is tra tio n is v a lid , and i f so,
F ig u re 3 : The re g is tra tio n d ia lo g it's added to th e c re d e n tia l array on th e user o b je c t. The

F ig u re 4 : Pass key in MacOS F ig u re 5 : Passkey in iOS

codem ag.com Passkey Authentication 19

 I n i t i a t i n g t h e Login s e q u e n c e
L is t in g 6 : kjihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA c o d e f o r v e r if y in g a n d a d d in g t h e c r e d e n tia l t o t h e u s e r
o b je c t c an b e se e n in L is t in g 5 . As ca n b e se e n in L i s t ­
i n g 5 , a s u c c e s s fu l r e g is t r a t io n is s e n t t o t h e c lie n t . I n
"challenge":
t h e re a l w o r ld , y o u 'll j u s t s a y " s u c c e s s fu l r e g is t r a t io n . "
"tim eout": 1800000,
"rp ld ": "localhost", T h e p u b lic k e y is s o m e th in g y o u ca n use t o v e r if y f u t u r e
"userverification ": "p re fe rre d ", a u t h e n t ic a tio n s . A lth o u g h y o u s h o u ld n 't j u s t s h a re i t w it h
"allowCredentials": [ t h e c lie n t o n t h e b ro w s e r, Le aking i t i s n 't a s e c u r ity is s u e .
{ T h is is b u i l t o n an a s y m m e tr ic k e y s ig n a tu r e ; as Long as
"c re d ld ": "wTRhXyFYEo3wl50xpg-XYZSyYDg",
t h e p r iv a te k e y is s a fe , y o u a re g o o d .
"publicKey":
"aaguid": "-_wwBxV0TsyMC24CBVfXvQ",
"prevCounter": 0, B e c a u s e t h e r e g is t r a t io n s u c c e e d e d , a p a s s k e y has n o w

"flags": {}. b e e n c re a te d f o r m e. I c an v e r if y t h i s in m y MacOS s y s te m

"type": "public-key" s e t t in g s , as ca n b e se e n in F ig u r e 4 .
}
], A d d it io n a lly , b e c a u s e t h is is a p a s s k e y , I ca n a ls o see i t
"authenticatorAttachment": "platform "
s y n c h e d t o m y iP h o n e in t h e sa m e iC lo u d a c c o u n t, as can
b e se e n in F ig u r e 5 .

T h is is t h e b e a u ty o f t h is s ta n d a rd . S ig n u p w a s easy, I

SPONSORED SIDEBAR c an use t h e c r e d e n tia l a n y w h e re , a n d th e r e 's n o p a s s w o rd

t o re m e m b e r. Y ay!
Ready to Modernize
a Legacy App?
Authentication
N eed a d v ic e o n m ig ra tin g
W ith r e g is t r a t io n d o n e , n o w Let's fo c u s o n a u t h e n t ic a t io n .
y e s te rd a y's le g a c y
a p p lic a tio n s to to d a y 's
To p e rfo rm a u t h e n t ic a t io n , t h e u s e r is e x p e c te d t o e n te r
m o d e rn p la tfo rm s ? Take
t h e i r u s e rn a m e , w h ic h th e n s e n d s o f f a c a ll t o t h e b e g i n -
a d v a n ta g e o f CODE
L o g in m e th o d . T h e b e g in L o g in m e th o d s im p ly c o m m u n i­
C o n s u ltin g 's years o f
c a te s t o t h e s e rv e r t h a t a c e r ta in u s e r is t r y in g t o s ig n in .
e x p e rie n c e a n d c o n ta c t us
to d a y to s c h e d u le a fre e
T h e s e rv e r is n o w e x p e c te d t o g e n e ra te a n d re m e m b e r a
c o n s u ltin g call t o discuss
y o u r o p tio n s . N o string s. c h a lle n g e , a n d f o r t h e u s e r t o s p e c ify c e r ta in a u t h e n t ic a ­

N o c o m m itm e n t. t io n r e q u ire m e n ts , w h ic h a re c o m m u n ic a te d b a c k t o t h e

N o th in g to buy. c lie n t , as ca n b e s e e n in L is t in g 6 . U s in g th e s e s p e c ific a ­

For m o re in fo rm a tio n , F ig u r e 6 : P asskey Login d ia lo g b o x o n MacOS t io n s c o m m u n ic a te d b y t h e s e rv e r, t h e c lie n t m a ke s a c a ll
v is it w w w .c o d e m a g .c o m / t o t h e n a v i g a t o r . c r e d e n t i a l s . g e t m e th o d . T h is s h o w s t h e
c o n s u ltin g o r e m a il us at u s e r a Login d ia lo g b o x , as c an b e se e n in F ig u r e 6 .
in fo @ c o d e m a g .c o m .
s a h il@ lo c a lh o s t.c o m R e g is te r L o g in
W h a t i f I 'd s a id " a u t h e n t ic a t o r A t t a c h m e n t " t o b e c ro s s -
p la tfo r m ? W e ll, t h e n y o u 'd b e p r o m p te d w it h a d if f e r e n t
" a u d it" : { d ia lo g b o x , o n e t h a t p r o m p ts y o u t o use NFC o r BLE, a n d
" v a lid E x p e c ta tio n s " : tru e , Lets y o u s c a n a QR c o d e , e tc ., t o c o m p le te a u t h e n t ic a t io n
" v a lid R e q u e s t" : tru e , o n a p h o n e . O r p e rh a p s a C h ro m e p r o f ile . N o te t h a t I'v e
" c o m p le te " : tru e , a ls o in c lu d e d t h e a l l o w C r e d e n t i a l s o b je c t t o L im it w h ic h
" jo u r n a l" : {},
c r e d e n tia ls m y RP is o k a y a c c e p tin g .
" w a r n in g " : {},
" in fo " : {}
T h e u s e r is n o w e x p e c te d t o c o m p le te t h e t o u c h lD o p e ra ­
h t i o n , w h ic h g e n e ra te s an a s s e rtio n re s p o n s e t h a t 's s e n t
" r e q u ir e d E x p e c ta tio n s " : {},
" o p tio n a lE x p e c ta tio n s " : {}, t o t h e s e rv e r in t h e e n d L o g in A P I c a ll. H ere t h e s e rv e r
" e x p e c ta tio n s " : {}, v a lid a te s t h e a s s e rtio n re c e iv e d w it h w h a t y o u e x p e c t.
" re q u e s t" : { T h is c an b e se e n in L is t in g 7 . A n d i f t h e a s s e rtio n s u c ­
"u s e rn a m e ": " s a h i l @ l o c a I h o s t . com " , c e e d s , y o u s e n d b a c k t h e a u t h e n t ic a t io n r e s u lt, as ca n be
" id " : {},
se e n in F ig u r e 7 .
" r a w ld " : {} ,
" ty p e " ; " p u b lic -k e y " ,
C o n g r a tu la tio n s ! Y ou j u s t r e g is te r e d a n d s ig n e d in u s in g
"re s p o n s e ": {
a pass key.
" a u th e n tic a to r D a ta " : " S Z Y N S Y g O jG h O N B c
" c lie n tD a ta J S O N " : " e y J 0 e X B H jo id 2 V iY >
" s ig n a tu r e " : " M E Q C IF lg y 6 fX U fB d w h a v u c
" u s e r H a n d le " : "1234" Summary
} P e rh a p s t h e e x p re s s io n " p a s s w o r d le s s " h a s b e e n o v e r ­
h u s e d in o u r in d u s tr y . W e 'v e s e e n m a jo r c o m p a n ie s use
" c lie n tD a ta " : {}»
m o b ile a u t h e n t ic a t o r a p p s a n d c a ll th e m p a s s w o rd le s s .
" a u th n rD a ta " : {}
O kay, t h e y w e re t e c h n ic a lly p a s s w o rd le s s , b u t i t i s n 't u n ­
t i l F ID 0 2 , t h a t w e h a v e t r u e p a s s w o rd r e p la c e m e n ts t h a t
F ig u r e 7 : S u c c e s s fu l a u t h e n t ic a t io n r e s u lt a re u n iv e r s a lly a c c e p te d a c ro s s t h e in d u s tr y . W ith t h e

20 Passkey Authentication codem ag.com

 L is t in g 7 : V a lid a tin g an a s s e rtio n

const user = usersfreq.body.username]

const assertionExpectations = {
challenge: req.session.challenge,
o rig in : env.RPOrigin,
fa cto r: 'e ith e r ',
publicKey: user.cre d e n tia ls[ 0 ] .publicKey,
prevCounter: u s e r.c re d e n tia ls ^] .prevCounter,
userHandle: req.body.response.userHandle
};
req.body.id =
coerceToArrayBuffer(req. body. id , " id " ) ;
req.body.rawld =
coerceToArrayBuffer(req.body.rawld, "ra w ld ");
const authnResult = await
f 21.assertionResult(
req.body, assertionExpectations);
re s .j son(authnResult);

a d v e n t o f p a s s k e y s , I f u l l y e x p e c t t h a t e v e ry m a jo r s it e
t h a t c a re s a b o u t s e c u r it y w i l l a d o p t i t as t h e s ta n d a r d
r e p la c e m e n t f o r p a s s w o rd s . A s u s e rs a n d o r g a n iz a tio n s
g e t c o m f o r t a b le w i t h t h e s ta n d a r d , I a ls o e x p e c t t h i s
t o g e t w id e ly a d o p te d in e n te r p r is e s in m o re s e c u re i m ­
p le m e n t a t io n s t h a t s u p p o r t a t t e s t a t io n a n d e n d - t o - e n d
s u p p ly c h a in s e c u r ity .

I t h i n k i t 's a ls o im p o r t a n t t o r e a liz e t h a t p a s s k e y s are a

p a s s w o rd re p la c e m e n t. T h e y a re b e t t e r th a n a n y p a s s w o rd
t h a t y o u ca n c o m e u p w i t h . U sed a lo n e , th e y 'r e s t i l l j u s t
a p a s s w o rd r e p la c e m e n t. W hen y o u re p la c e th e m w it h
a h a rd w a re -b a s e d a u t h e n t ic a to r , y o u 'r e a lre a d y u s in g a
m u c h m o re s e c u re m e c h a n is m t h a t c a n n o t b e s c a le d in
an a tta c k . R e m e m b e r t h a t p a s s k e y s ca n b e s h a re d v ia A D V E R TISER S IN D E X
iC lo u d f a m ily s h a r in g , a n d y o u r p a s s k e y is j u s t as s e c u re
as y o u r iC lo u d a c c o u n t. I n c o n tr a s t, h a rd w a re k e y s h a r in g
is n o t an a t t a c k t h a t ca n s c a le e a s ily a n d c e r t a in ly c a n ­ Advertisers Index
n o t b e s h a re d a c ro s s t h e in t e r n e t . A d d it io n a lly c e r ta in
h a rd w a re keys n o w s u p p o r t b io m e tr ic s o r e n t e r in g p in s .
C O D E C o n s u ltin g - - A I S e rv ic e s
T h is m ak e s th e m e q u iv a le n t t o m u l t i - f a c t o r a u t h e n t ic a ­
w w w .c o d e m a g .c o m /a i-s e rv ic e s 2
t i o n , s o m e th in g y o u h a v e — t h e k e y — a n d s o m e th in g y o u
k n o w — t h e p in . C O D E C o n s u ltin g La nguage M odels CBA
f

w w w .c o d e m a g .c o m /C o d e 75
H o w e v e r, n o t h in g s to p s y o u fro m u s in g p a s s k e y as o n e o f
C O D E S ta ffin g
t h e fa c to r s in a u t h e n t ic a t io n . R e a lly , w h a t s to p s y o u fro m
w w w .c o d e m a g .c o m /s ta ffin g 76
a c c e p tin g a us e r's p a s s k e y a n d t h e n p r o m p tin g th e m fro m
an MFA p r o m p t o n t h e i r m o b ile p h o n e u s in g an O A u th - d tS e a rc h
c o m p a tib le app? I s n 't t h is m u lt i- f a c t o r , w h e re y o u 'v e w w w .d tS e a rc h .c o m 15
re p la c e d o n e o f t h e w e a k fa c to r s , t h e p a s s w o rd , w it h a
D e v ln te r s e c tio n
m u c h m o re s e c u re re p la c e m e n t, w h ic h is p a ss ke y?
w w w .d e v in te r s e c tio n .c o m 39
Advertising Sales:
Im p le m e n ta t io n s lik e t h i s a re g o in g t o o ffe r user c o n ­ LE A D T e c h n o lo g ie s Tam m y Ferguson
8 3 2 -7 1 7 -4 4 4 5 e x t 26
v e n ie n c e a n d g r e a te r s e c u r ity in o n e s w o o p . T h is is t h e w w w .L e a d to o ls .c o m 5
ta m m y @ c o d e m a g .c o m
h o ly g r a il o f s e c u r ity w h e re y o u p ro v id e s o lu tio n s t h a t are
R aven DB
c o n v e n ie n t, so us e rs d o n 't w o r k a ro u n d th e m , a n d t h a t
w w w .r a v e n d b .n e t 7
a re s e c u re b y d e s ig n . T h is is w h y I f e e l t h e s ta n d a rd w i l l
g a in u n iv e r s a l a c c e p ta n c e a n d w h y t h e e n t ir e in d u s tr y is
b e h in d i t .

I h o p e y o u fo u n d t h i s a r t ic le u s e fu l. I k n o w t h is w a s a
l i t t l e b i t c o d e h e a v y , b u t w e 're d e v e lo p e rs a n d w e Like t o
see c o d e , r ig h t?

T his Listing is p ro v id e d as a c o u rte s y

U n t il n e x t tim e , s e c u re c o d in g . t o o u r readers and a d v e rtis e rs .
The p u b lis h e r assum es no re s p o n s i­

Sa h it M a lik b i li t y fo r errors o r o m is sion s.

CODE

codem ag.com Passkey Authentication 21

You're M issing O u t on
O p e n -S o u rc e LLMs!
By now, Km certain, you tried one of the many online services that allow you to
use a Large Language Model. If so, you might even be familiar with responses
like, "I'm sorry, but I cannot provide assistance on ..." In this practical guide,
I will talk about why open-source language models are important, why you'd
want to use them, how to choose and where to find a model, and how to run
them on your own local or hosted machine.
 Let's be clear, c ra ftin g n a rra tiv e s is n 't m y fo r te — hence T his is n o t o n ly im p o rta n t from an e n g in e e rin g s ta n d ­
m y ro le as a s o ftw a re d e v e lo p e r and n o t th e n e x t T o lk­ p o in t (w e do Like to tin k e r a fte r a ll) , b u t w ith th e re­
ie n . B u t I do e n jo y h ig h fa n ta s y and I ' l l a d m it t h a t I'v e lease o f research papers a lo n g sid e m odels and tra in in g
s p e n t m any a n ig h t p la y in g D ungeons and Dragons w ith da ta se ts, th e c o m m u n ity has picked up on m any o f the se
m y frie n d s . You can im a g in e how e x c ite d I was w hen I de ve lo p m e n ts and im p ro v e d on th e m . I'd be g e n u in e ly
discove red ChatGPT in e a rly 20 2 3, p ro m is in g an u n e n d in g su rp ris ed i f th e c u rre n t s ta te -o f- th e - a r t m odels released
w e ll o f ideas fo r cha ra cters, v illa in s , and s tic k y s itu a tio n s as paid o ffe rin g s w o u ld be as advanced as th e y are i f th e
fo r m y p la ye rs' in tre p id heroes to g e t o u t o f. ope n-so urce c o m m u n ity d id n 't e x is t.

Philipp Bauer U n til one day, I tr ie d to create some mean ga ngsters to

p b a u e r@ c o d e m a g .c o m a ccos t m y players in a b ig c ity and i t o n ly said, " I'm Open-source proves, yet again, how
sorry, b u t as an A l language m odel, i t is n o t a p p ro p ria te

I
P h ilipp is an accom plished
to g lo r ify c rim in a l b eh av io r, even in a fic tio n a l s e ttin g ."
senio r softw are developer important freely accessible software is.
I pleaded w ith th e A l, tr ie d to g e t aro und its s a fe ty m ea­
w ith over 15 years o f profes­
sures, and cried o u t to th e s ilic o n gods! I t k e p t in s is tin g
s io na l experience in w ebsite
t h a t we m us t be nice to each othe r.
and a p p lic a tio n develop­
m ent. W ith an edu ca tio n al
background in c om p ute r s ci­ Open source also pre ven ts a fe w b ig players from c a p tu r­
ences and media design, he in g th e te c h n o lo g y and d ic ta tin g w h a t LLMs lo o k Like.
has honed his skills in f u ll­ ChatGPT kept insisting that we The m o tiv e s and p rio ritie s o f a Large com pa ny o fte n d o n 't
stack developm ent, server a lig n w ith th o s e o f th e peo ple, a fte r a ll. A de m oc ra tiz ed
must be nice to each other.
a d m in is tra tio n , and netw ork landscape w ith o p e n ly accessible to o ls evens o u t th e
m anagem ent. P hilipp's keen p la y in g fie ld , even i f th e re is a lim ita tio n based on th e
eye fo r design translates c o m p u tin g p ow e r th a t's a v a ila b le fo r tr a in in g fo u n d a tio n
in to user inte rface s th a t are m odels.
b o th fu n c tio n a l and visu ally D efeated and d is a p p o in te d , I th o u g h t a b o u t how to crack
appealing. open t h a t v a u lt o f c re a tiv ity in m y head. My im a g in a ry
Y o u H a d M y C u rio s ity ,
b lo w to rc h e s and th ie f's to o ls were no m atch fo r hardened
P h ilip p e x p e rim e n ts w ith
s te e l. b u t N o w Y o u H a v e M y A t t e n t io n
v a rio u s ways o f im p le m e n t­
I f y o u have th e im p re ssio n t h a t using ope n-so urce LLMs
in g e m e rg in g A l te c h n o l­
Then I discove red t h a t som e v e ry s m a rt pe op le had is m ore c o m p lic a te d th a n o p e n in g a w e b s ite and as king a
ogy, Like ChatGPT and Lo­
tra in e d o pe n-so urce language m odels t h a t anyone co uld q u e s tio n , y o u 're a b s o lu te ly r ig h t! D o n 't Let t h a t scare you
c a lly e x e c u ta b le LLMs and
run w ith th e r ig h t hardw are and a p p lic a tio n s . F inally, th e aw ay th o u g h , it's w e ll w o rth u n d e rs ta n d in g th e fo llo w in g
s p e e c h -to -te x t m odels lik e
O pe nA I W hisp er and V e cto r pro m ise o f u n re s tric te d m odels t h a t d o n 't t e ll y o u w h a t te rm s, ideas, and p rin c ip le s .
DBs, to crea te new and y o u can and c a n 't do! In te re s tin g and e v il v illa in s may
e x c itin g ways o f in te r a c t­ te rro riz e th e fre e c itiz e n s o f th e Sword Coast y e t again I ' l l s ta rt by lo o k in g a t som e o f th e reasons w hy y o u , as
in g w ith so ftw a re . and m y cam pa ign Lives o n! a d e ve lo p e r o r o rg a n iz a tio n , m ig h t w a n t to use an op e n-
source LLM o ver a paid o ffe rin g by O penA I o r M ic ro s o ft.
A stro n g advocate fo r th e A fte rw a rd , I ' l l discuss th e ty p e s o f m odels t h a t e x is t, th e
use o f open-source s o ft­
It's D a n g e r o u s t o G o A lo n e :
v a ria tio n s th e y m ay com e in , and w here to fin d and how
ware, he's c o n trib u te d to A C o lla b o r a tiv e J o u r n e y to choose m odels. I'Ll co nclude by p re s e n tin g some ways
several p ro je c ts , in c lu d in g In th e fo llo w in g m on th s, I clo se ly fo llo w e d th e news to g e t s ta rte d ru n n in g LLMs using a fe w d iffe r e n t a p p lic a ­
P h o tin o , a cro ss -p la tfo rm and upd ate s s u rro u n d in g th is very a c tiv e c o m m u n ity o f tio n s y o u can d o w n lo a d and use fo r free.
Library fo r .NET t h a t enables
s o ftw a re en gine ers and da ta s c ie n tis ts w ith w o n d e r and
developers to b u ild n a tiv e
a d m ira tio n , and a good m easure o f p e rp le x ity due to th e
user in te rfa c e s fo r W indows,
new co nc ep ts and Lingo s u rro u n d in g a ll o f i t . I t m o tiv a te d
W e D o W h a t W e M u s t B ecause W e Can
macOS, and Linux.
me to learn m ore. I lis te n e d to A nd re j K arpath y e x p la in in g As w ith th e a d o p tio n o f any o th e r new te c h n o lo g y , you
how to b u ild GPT, learned w h a t d iffe re n c e tr a in in g data m ust be able to eva luate w h e th e r it's th e r ig h t one fo r y o u r
and m e th o d o lo g ie s make, and w h a t a ll th es e param eters use case. Using so m e th in g ju s t because you can is rarely
beyond "te m p e ra tu re " m ean. B u t m ain ly , I Learned how to a good idea. You m ig h t Look a t these argum ents and con ­
s ta rt using th is am azing te c h n o lo g y in m y ow n w o rk . And clude th a t a SaaS o ffe rin g is serving you ju s t fin e . Be aware
in th e p ro g ra m m in g Language t h a t I Like th e m os t (w h ic h th a t a lth o u g h th e fo llo w in g are im p o rta n t co nsid erations
is C#, o f cou rs e). to keep in m in d , th e y are, by far, n o t th e o n ly ones.

A lo t has happened in th e o pe n-so urce LLM space since

th e n , m ore th a n I co uld p u t dow n in ju s t one a rtic le . And Using som ething just because
w ith th e c u rre n t speed o f change, i t w o u ld lik e ly be dated

I
by th e tim e th is was p rin te d and read by anyone anyway.
One th in g is c le a r to me th o u g h : Open source proves y e t
you can is rarely a good idea.CBA
aga in how im p o rta n t fre e ly accessible s o ftw a re is . W ith
th e gra cio us s u p p o rt o f large co m p anie s Like M eta and
M ic ro s o ft, and s m a lle r players lik e M is tra L A I, e t a l., w ho C o n tro l
a ll released fo u n d a tio n m odels, th e c o m m u n ity fin e -tu n e s As w ith any SaaS o ffe rin g o u t th e re , it's o u t o f y o u r c o n ­
and d is trib u te s m odels fo r s p e c ific use cases and o ffe rs an t r o l w h e th e r th e b e h a v io r o f fe a tu re s changes. This is
a lte rn a tiv e to th e closed -source, paid m odels t h a t O penA I e s p e c ia lly tru e fo r th e e v e r-e v o lv in g LLMs and th e ex act
o r G oogle offe r. th in g t h a t drove m y a tte n tio n to ope n-so urce LLMs.

You're M issing O u t o n O p en -S ou rce LLMs! codem ag.com

SaaS o ffe rin g s m us t ev olve w ith th e a tta c k s leveraged P r iv a c y a n d C o m p lia n c e
a g a in s t th e m and th e ir s a fe ty a lig n m e n t m us t be upd ated The s a fe ty o f cu s to m e r data and in te lle c tu a l p ro p e rty is
to th w a r t new p ro m p tin g a tta c k s t h a t expose dangerous s t i l l a b ig issue fo r m any w hen i t com es to using LLMs
and u nw a n te d b eh av io r. They also m us t im p ro v e p e rfo r­ as a service. I t was th e m ain concern I heard w hen t a lk ­
m ance th ro u g h c o n tin u e d fin e - tu n in g o f th e ir m odels to in g to pe op le a b o u t us ing LLMs a t 2023's D E V intersection
p re v e n t fa llin g b e h in d th e c o m p e titio n . con fe re nc e. I un d ersta nd w here th e uneasiness comes
fro m . E specially w hen y o u co ns id er u p lo a d in g in te rn a l
These kinds o f tw ea ks can in flu e n c e th e o v e ra ll b e h a v io r docum en ts fo r processing and re trie v a l fo r u sin g , fo r
o f th e m od el and break th e w o rk flo w s t h a t y o u im p le ­ exam ple, R e trie v a l A u g m e nted G eneration (RAG) a p p li­
m en ted on y o u r end, re q u irin g y o u o r y o u r tea m to c o n ­ c a tio n s , w he re in th e p ro s p e c t o f h a vin g s e n s itiv e data
s ta n tly e v a lu a te and up d ate th e p ro m p ts y o u use in y o u r " o u t th e re " d o e s n 't s it w e ll. Data breaches seem to be a
a p p lic a tio n . m a tte r o f "w h e n " and n o t " i f ' th es e days, and th is can be
th e s in g u la r issue t h a t excludes a SaaS o ffe rin g fo r some
I d o n 't ju s t cla im th is based on m y ow n o b s e rv a tio n s . A com pa nies .
s tu d y calle d "H o w Is ChatGPT's B e h a vio r C hanging o ve r
T im e?" released by a te am a t S ta n fo rd U n iv e rs ity and UC A n o th e r issue m ay be co m p lia n ce agree m ents. They may
B erkeley d e m o n stra te d th is be h av io r. They asked a s e t e ith e r be a b ig th re s h o ld fo r a d o p tio n o r ru le o u t using
o f q ue s tio n s in March 2023 and th e n in Ju ne 2 02 3. The th ir d - p a r ty p ro vide rs a lto g e th e r. I f y o u r ow n e n v iro n m e n t
answers were recorded and com pared, sh o w in g s ig n ific a n t is alre ady s e t up to be c o m p lia n t, i t m ig h t ju s t be easier
d iffe re n c e s be tw een th e m o d el versio ns. They co nclu ded to use a lo c a lly run o pe n-so urce LLM. I t m ig h t also be
th a t: th e o n ly w ay to s ta y c o m p lia n t w ith lo c a l Laws t h a t fo r ­
b id y o u from s e nd in g user da ta in to o th e r ju ris d ic tio n s ,
"O ur fin d in g s dem onstrate th a t th e beha vio r o f lik e th e U n ite d S tates ( th in k GDPR), e ith e r by using y o u r
GPT-3.5 and GPT-4 has varied s ig n ific a n tly over ow n servers o r h o s tin g a m od el w ith y o u r lo c a l d a ta c e n te r
a re la tiv e ly s h o rt am ou nt o f tim e. This h ig h lig h ts pro vide r.
th e need to co ntin uo u sly evaluate and assess th e
behavior o f LLM d rifts in app lic atio ns , especially O f f lin e U s a g e
as i t is n o t tran sp aren t how LLMs such as ChatGPT T his la s t one is b o th an a rg u m e n t fo r re s ilie n c e o f y o u r
are updated over tim e . [ . . . ] Im p ro v in g th e m od­ a p p lic a tio n s and an e x c ite d lo o k in to th e fu tu re o f how
el's perform ance on some tasks, fo r exam ple w ith we use LLMs as everyday a ssis ta n ts .
fin e -tu n in g on a d d itio n a l data, can have unex­
pected side effects on its beha vio r in o th e r tas ks." It's an enorm ously d if fic u lt ta sk to provide c o n s is te n t ser­
( h ttp s ://a rx iv .o rg /a b s /2 3 0 7 .0 9 0 0 9 ) vice to m illio n s o f people using LLMs in th e cloud. The in ­
fra s tru c tu re needed fo r ju s t th is s in g u la r service is massive.
To re ite ra te , fin e - tu n in g to im p ro v e pe rfo rm ance and u p ­ And w ith scale and co m p le xity, th e Likelihood o f outages
d a tin g m od el a lig n m e n t is good and necessary w o rk ! B u t grows to o . This is tru e fo r any SaaS o ffe rin g , o f course, and
s a fe ty a lig n m e n t, e sp ec ia lly , is a to u g h issue to solve and so th e same con sid erations m us t be made fo r LLM services.
hard to g e t r ig h t fo r everyone. You m ig h t end up w ith a For m ost use-cases, it's b es t pra ctic e to design y o u r a p p li­
use case t h a t w orks a t f ir s t b u t th e n breaks o ve r tim e ca tio n w ith resilience in m in d, t h a t is , o ffe rin g a degraded
because it's deem ed dangerous o r has u nw a n te d b e h a v io r fe a tu re s e t and cache requests to th e a ffe c te d service fo r
even th o u g h it's c o m p le te ly le g itim a te . later. F ailover to a n o th e r region is a n o th e r good w ay to
handle service in te rru p tio n s where possible.

You m ig h t have a use-case, th o u g h , t h a t requires u n ­

You will be responsible for in te rru p te d a v a ila b ility . T his m ay be e s p e c ia lly tru e fo r
a p p lic a tio n s o u t in th e fie ld w here c e llu la r service is u n ­
providing a certain level of
a v a ila b le (w elcom e to ru ra l A m erica) o r i f y o u have an ap ­
safety and alignm ent. p lic a tio n t h a t needs to be a v a ila b le in d is a s te r scenarios.
W ith LLMs t h a t can run on c o m m o d ity hardw are, th is is
a p o s s ib ility to d a y . C h a ttin g w ith an LLM d u rin g a re c e n t
f lig h t w ith o u t in te r n e t access was a re a l h e a d -tu rn e r!
O pen-sou rce LLMs d o n 't ch ange once th e y 're d o w n lo a d ­
ed, o f course . A nd y o u can e a s ily fin d m odels t h a t are u n ­
censored and have no q ua rre ls re s p o n d in g to any and a ll
o f y o u r q u e s tio n s . There are also m odels t h a t com e w ith Small m odels will make
a c e rta in le v e l o f p re -tra in e d s a fe ty . These m odels a re n 't
the personal assistants on our
g u a ra n te e d t o be as e ffe c tiv e a t re s is tin g a tta c k s , lik e
th e in fa m o u s DAN (Do A n y th in g N ow ) ja ilb r e a k , and show phones incredibly powerful.
t h a t o p e n -s o u rc e LLMs can be a d o u b le -e d g e d sw ord . On
th e one ha nd , y o u 're n o t Lim ite d by th e s a fe ty a lig n m e n t
o f th e Large SaaS p ro v id e rs . On th e o th e r ha nd , y o u 'll be
re s p o n s ib le fo r p ro v id in g a c e rta in le v e l o f s a fe ty and A nd th is is w h y I g e t e x c ite d fo r th e fu tu re o f lo c a lly run
a lig n m e n t t h a t y o u o r y o u r o rg a n iz a tio n can be co m ­ LLMs. S m a ll m odels becom e m ore and m ore capable and
fo rta b le w ith . E sp ec ia lly w hen th e p ro d u c t y o u im p le ­ w ill make th e p e rson a l as sis tan ts on o u r phones in c re d ­
m e n t an o p e n -s o u rc e LLM in t o is m ade a v a ila b le to th e ib ly p o w e rfu l, w h ile keep ing o u r p riv a te in fo rm a tio n o u t
p u b lic ! o f th e hands o f d ata brokers.

codem ag.com You're M issing O u t o n O pe n-S o urce LLMs! 25

 Additional ConsiderationskjihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA
The am azing fa c t a b o u t fin e - tu n in g m odels is t h a t it's fa r
B eyond th es e th re e co n s id e ra tio n s , y o u m ig h t also fin d less re s o u rc e -in te n s iv e th a n c re a tin g a fo u n d a tio n m odel.
t h a t using o pe n-so urce m odels on y o u r ow n hardw are can It's q u ite a c hie vab le fo r in d iv id u a ls and com pa nies to
save y o u m oney, e s p e c ia lly w hen a SaaS o ffe rin g is based tra in a fo u n d a tio n m od el on th e s p e c ific use case th e y
on a pe r-u se r basis and an e q u iv a le n t o pe n-so urce o p tio n w a n t to use i t for.
is a v a ila b le to y o u . You m ay also co ns id er fin e - tu n in g a
m od el on y o u r ow n d a ta , and, a lth o u g h th is is possib le Besides th e c h a t-s ty le be ha vior, th e n e x t m os t com m on
using O penA I and Azure O penA I, da ta p riv a c y and c o m p li­ fin e -tu n e y o u can fin d is fo r in s tru c tio n s . These m odels
ance issues m ig h t be an issue here as w e ll. Then th e re are are tra in e d to g iv e answers and adhere to an in s tru c tio n
d e p lo y m e n t c o n s id e ra tio n s , w h ic h is a h ig h ly in d iv id u a l give n to th e m . The n o ta b le d iffe re n c e is t h a t c h a t m od­
and use-case d rive n issue t h a t w ill have to be analyzed els m ay c o n tin u e to e la b o ra te , w hen ta sk ed , to produce
based on y o u r needs. a s p e c ific o u tp u t lik e , fo r exam ple, "y e s -" o r " n o - " o n ly
answ ers. In s tru c tio n fin e -tu n e d m odels ty p ic a lly adhere
m ore to th e g iv e n re s tric tio n .
N o t A ll M o d e ls A re C reated Equal
When y o u lo o k a t th e t e x t g e n e ra tio n m od el c a ta lo g u e on Navigating Models on Hugging Face
H ug g in g Face (h u g g in g fa c e .c o ) y o u m ig h t w o n d e r how to Llam a, A lp aca , P laty pus, Orca— a ll o f th es e and o th e rs
choose fro m th e o ve r 3 7 ,0 0 0 m odels t h a t are c u rre n tly re fe r to m odels, tr a in in g d ata se ts, o r te c h n iq u e s used
a v a ila b le . I rem em ber fe e lin g q u ite ove rw he lm ed a t f ir s t to create a d a ta s e t. Each m od el on H ug ging Face has a
and w a s n 't sure w here to b e g in . A fte r u n d e rs ta n d in g th e "m o d e l c a rd " t h a t in c lu d e s th es e d e ta ils and m ore. An im ­
Build local LLM apps process o f how th es e m odels are cre ate d and w here to p o rta n t piece o f in fo rm a tio n to Look o u t fo r is th e p ro m p t
in .NET fin d good com pa riso ns , i t was easy to fin d th e r ig h t m od ­ te m p la te t h a t th e m od el was tra in e d on . These te m p la te s
e l fo r m y use cases. separate th e system , user, and a s s is ta n t messages from
This is the first part of a series
each o th e r. W ith th is , th e m od el is n 't g e ttin g confus ed
of articles. The next article will
focus on building applications Building a Strong Foundation a b o u t w ho said w h a t and can c o rre c tly in fe r fro m p re v i­

in C# and .NET that can use an I t a ll s ta rts w ith a fo u n d a tio n m odel. These are m odels ous co n v e rs a tio n s . I f no s p e c ific te m p la te is m e n tio n e d ,
open-source LLM directly on tra in e d by o rg a n iz a tio n s t h a t can a ffo rd to buy o r re n t th e te m p la te o f th e m odel's fo u n d a tio n m od el is us ua lly
your machine instead of using large am o un ts o f hardw are and pay fo r th e necessary a safe b e t.
an external API. Look out for steps t h a t com e before th e a c tu a l tr a in in g .
that in another issue of CODE Comparing Llamas and Alpacas
Magazine! Let's ta k e M eta's p o p u la r Llama 2 m od el as an exam ple. LLMs are fre q u e n tly te s te d a g a in s t m a nua l and a u to m a te d
I t was tra in e d on an im m ense a m o u n t o f 2 t r illio n to ken s s u ite s o f te s ts t h a t can g iv e us a rough idea o f th e ir ca­
(ro u g h ly 500 b illio n w o rd s ). These t e x t sources needed to p a b ilitie s .
be c o lle c te d fro m va rio u s sources and th e n pre-processed
before th e y co u ld be used to tra in th e m ode l. The data The "Open LLM Leaderboard" on H ugging Face is a decent
used fo r th is tra in in g is la rg e ly u n s tru c tu re d and u n la - fir s t place to g e t starte d in th e search fo r an LLM (see
bele d te x t. The tr a in in g o f th e th re e m o d el sizes o f 7, 13, h ttp s://b it.L y /llm L e a d e rb o a rd ). I t lets you f ilt e r by m odel
and 70 b illio n param eters (m ore on t h a t Later) t h a t M eta type , num ber o f param eters (size), and precision, and you
released to o k a co m b in e d 3.3 m illio n GPU hours. can s o rt by an average score or sp ecific te s t scores. Be aware
th a t there's some critic is m around th is leaderboard because
there are models t h a t tr y to game th e tests by in c lu d in g both
The training took a combined th e questions and th e answers in th e ir tra in in g data. The
c o m m un ity flags these models fo r th is beha vio r and excludes

I
them by d efa u lt. B u t there's s t ill a tre nd away from th is lead­
3.3 million GPU hours. erboard as a reliable resource a t th e m om ent.

A n o th e r in te re s tin g Leaderboard can be fo u n d on th e

C h a tb o t A rena w e b s ite ( h ttp s ://a re n a .lm s y s .o rg ). C hat­
A t th is p o in t, th e m od el has a lo t o f know led ge, b u t i t b o t A rena generates its ra n kin g s by p it t in g tw o random
d o e s n 't q u ite know w h a t to do w ith i t . I f y o u asked i t a LLMs a g a in s t each o th e r t h a t b o th g e t th e same user
q u e s tio n i t w o u ld s tru g g le to g iv e y o u a c o h e re n t answer. p ro m p t. Users d o n 't know w h ic h m odels are re p ly in g . They
T his is w here th e n e x t tr a in in g step com es in . can th e n v o te fo r w h ic h m o d el o u tp u t th e y lik e b e tte r.
Votes are a gg re g ate d in t o a score on th e Leaderboard.
Learning the Ropes Since th is is n o t a u n ifo rm te s t t h a t is a p p lie d to a ll m od ­
A fo u n d a tio n (o r p re -tra in e d ) m od el needs to Learn how els equ ally , i t relies on th e average s u b je c tiv e experien ce
it's ex pe cted to behave w hen th e user asks a q u e s tio n . o f th e crow d. I t can o n ly g iv e a g e n era l com pa riso n b u t
The m o s t fa m ilia r use case r ig h t now is a c h a t-s ty le be ­ c a n 't disce rn th e s tre n g th s and weaknesses o f a m odel
havio r. To Learn how to have a p ro p e r c o n v e rs a tio n , a (e. g ., w r itin g vs. c o d in g ).
m od el is tra in e d on cu ra te d c o n v e rs a tio n a l datas ets.
The d atas ets fo r th is sup ervise d fin e - tu n in g step are The b e s t source fo r m od el co m p arison s is th e "LocaLL-
lab eled to c o m m u n ic a te th e desired o u tc o m e fo r each LaMA" s u b re d d it ( h ttp s ://re d d it.e o m /r/L o c a L L L a M A ).
in s ta n c e o f in p u t d ata . F urth er R ein fo rc e m e n t Learning S earching fo r "c o m p a ris o n " in th is sub w ill re tu rn several
w ith Human Feedback (RLHF) and o th e r te c h n iq u e s can users' ow n co m p aris on re su lts . It's w o rth lo o k in g a t the se
be used as a n o th e r step to im p ro v e th e o u tp u t q u a lity because th e y o fte n g iv e a d e ta ile d d e s c rip tio n o f th e re­
and a lig n th e m od el a cco rd in g to b e h a v io r and s a fe ty s u lts and a re n 't as s u s c e p tib le to m a n ip u la tio n th ro u g h
re q uire m e nts. tr a in in g on te s t q ue s tio n s as th e s ta nd a rd iz ed te s ts o f

26 You're M issing O u t o n O p en -S ou rce LLMs! codem ag.com

th e Open LLM Leaderboard. T his is also th e place to fin d d e p e n d e n t on th e p re cis io n o f th e m od el w e ig h ts , its ar­
ge n era l in fo rm a tio n a b o u t LLMs and th e la te s t news and c h ite c tu re , and th e le n g th o f th e user in p u t.
de ve lo p m e n ts in th e fie ld .
There is a w ay to reduce th e m em ory requirem ents, th o u g h .
In th e end, it's alw ays im p o rta n t to te s t th e chosen m o de l
on th e in te n d e d use case. Accelerating LLMs with Quantization
Q u an tiz a tio n is a te c h n iq u e th a t aim s to m in im iz e th e nu ­
m eric al precision required fo r w e ig h ts and a c tiv a tio n s w h ile
B ring Y our O w n M a c h in e preserving th e o v e ra ll fu n c tio n a lity o f an LLM. This ap­
Now y o u know w h a t th e d iffe r e n t ty p e s o f m odels are, proach enables s ig n ific a n t enhancem ents in c o m p u ta tio n a l
w here to fin d th e m , and how to choose one based on y o u r e ffic ie n cy, m em ory savings, and reduced energy consum p­
use-case and o p e n ly accessible co m p arison s. W ith t h a t tio n . As a re su lt, qua ntize d m odels are h ig h ly b e n e fic ia l
o u t o f th e way, I can tu rn to th e m ore p ra c tic a l p o rtio n fo r d e p lo yin g LLMs on devices w ith lim ite d resources such
in th e q u e st to run LLMs. In th is s e c tio n o f th e a rtic le , as edge c o m p u tin g o r m ob ile devices. However, these be n ­
I ' l l ta lk a b o u t hardw are re q uire m e nts and (c u rre n tly ) su p ­ e fits may be accom panied by Lim ita tio ns , such as loss o f
p o rte d hardw are and th e s o ftw a re needed to run an LLM. pre cisio n , w h ich could re s u lt in s lig h t accuracy reductions
com pared to th e o rig in a l h ig h -p re c is io n m odels.
I ' l l be g in by lo o k in g a t th e d is k and m em ory size re q u ire ­
m ents o f LLMs and a te c h n iq u e ca lle d q u a n tiz a tio n t h a t In m os t cases, th e b e n e fit o f lo w e r hardw are re q uire m e nts
reduces th es e re q uire m e nts. beats th e s lig h t re d u c tio n in m od el pe rfo rm ance. In s te a d
o f using 1 6 - b it flo a tin g p o in ts per w e ig h t and a c tiv a ­
One Size Does Not Fit All tio n s , a q u a n tiz e d m od el m ay use 8 - b it in te g e rs , a ll th e
When d is cu ss in g LLM sizes, it's c ru c ia l to un d ersta nd t h a t w ay dow n to 2 - b it in te g e rs . The lo w e r th e p re c is io n , th e
"s iz e " refers to th e nu m be r o f param eters a s p e c ific m o de l h ig h e r th e im p a c t on th e m o del, th o u g h .
c o n ta in s . Param eters in LLMs are a d ju s ta b le co m p on e nts
o r w e ig h ts t h a t in flu e n c e b e h a v io r and pe rfo rm ance d u r­ The 8 - b it and 5 - b it (lik e th e m ixed 5_K_M q u a n t) are th e
in g tasks, such as t e x t g e n e ra tio n and tra n s la tio n . They m os t p o p u la r o p tio n s fo r q u a n tiz a tio n th es e days. You can
re p re se nt c o n n e c tio n s be tw een neurons in a n e u ra l n e t­ change th e fo rm u la fro m above s lig h tly to c a lc u la te th e
w ork a rc h ite c tu re and enable i t to learn com plex re la tio n ­ size o f q u a n tiz e d m odels.
sh ip s and p a tte rn s w ith in Language.
The fo rm u la fo r size in GB is : # o f p a r a m e te r s (b y t e s ) / 8
In th e landscape o f LLMs, size is a s ig n ific a n t fa c to r in ( b i t ) * q u a n t s iz e ( in b i t ) / ( 1 0 0 0 A 3 ) = x GB
d e te rm in in g th e ir c a p a b ilitie s . A lth o u g h i t m ay seem t h a t
th e Larger m od el alw ays perfo rm s b e tte r, s e le c tin g th e ap ­ For a 7B pa ra m e te r m od el w ith 5 - b it (5_K _M , ~ 5 .6 7 b i t /
p ro p ria te size fo r y o u r use case requires c a re fu l co n sid e r­ w e ig h t) q u a n tiz a tio n , th is m eans: ( 7 * 1 0 A 9 ) / 8 * 5 .6 7
a tio n . Larger m odels re qu ire m ore m em ory and processing / ( 1 0 0 0 A 3 ) = 4 . 6 2 GB
pow er. S m aller m odels m ay be s u ffic ie n t fo r y o u r use case
w h ile b e in g able to d e liv e r fa s te r response tim e s on a Com pared to th e 1 6 - b it m odel, th is is ro u g h ly a th re e -
s m a lle r o p e ra tio n a l b u d g e t. tim e s re d u c tio n in size!

The ge n era l ru le is t h a t Larger m odels te n d to excel a t Q uantiz ed m odels com e as GGUF (m a in ly CPU/macOS
ta c k lin g m ore com plex tasks and are s u ita b le fo r c re a tiv e use), AWQ, and GPTQ ( 4 b it / 8 b it GPU use) fo rm a ts . Tom
w r itin g a p p lic a tio n s due to th e ir a b ilit y to make in tric a te J o b b in s , also know n in th e c o m m u n ity as TheB loke, is
c o n n e c tio n s th ro u g h th e ir v a s t nu m be r o f param eters and a w e ll-re c o g n iz e d c o n tr ib u to r w ho d is trib u te s q u a n tiz e d
layers. However, s m a ll y e t p o w e rfu l a lte rn a tiv e s sh ou ld ve rsion s o f m odels on H ug g in g Face ( h ttp s ://h u g g in g -
n o t be o ve rlo o ke d , as th e y o fte n surpass e x p e c ta tio n s face.co/T heB Loke).
by d e liv e rin g re su lts co m p ara ble to th o s e gen erated by
la rg er c o u n te rp a rts fo r c e rta in a p p lic a tio n s . Hardware Requirements for Local Inference
One, i f n o t th e b ig g e s t, fa c to r fo r th e hardw are req uire d
Generally, y o u 'll fin d m odels ran ging from 7 B (illio n ) to 70B fo r in fe re n c e using LLMs is th e a v a ila b le m em ory b a n d ­
param eters w ith exceptio ns be lo w and above these sizes. w id th betw e en th e m em ory and processor o f th e system .
T ypical co unts are 7B, 13B, 33B , 65B, and 70B param eters. The large n u m be r o f m a th e m a tic a l o p e ra tio n s perform e d
d u rin g in fe re n c e dem and fre q u e n t re a din g and w r itin g o f
The re qu ired size (in b y te s ) on d is k can be ro u g h ly ca lc u ­ data in to th e m em ory.
la te d by m u ltip ly in g th e pa ra m e te r size by tw o , because
each pa ra m e te r requires tw o by tes to rep re se nt a param ­ A lth o u g h it's p os sible to run in fe re n c e on CPU and
e te r as a 1 6 - b it flo a tin g p o in t num ber. RAM, it's n o t th e p re fe rre d w ay because th e b a n d w id th
pe r DIMM is Lim ited to a m axim um o f 64G B /s fo r DDR5
The fo rm u la fo r th e size in GB is : # jihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA
o f p a r a m e te r s * 2 RAM and th e t o t a l b a n d w id th is capped by th e proces­
( b y t e ) / ( 1 0 0 0 A 3 ) = x GB sor (e. g ., 8 9 .6 G B /s fo r th e c u rre n t I n t e l i9 1 4 9 00 K ). A
m odern consum er g ra ph ic s card, lik e th e N VID IA GeForce
A
For a 7B pa ra m e te r m o del, th is m eans: ( 7 * 1 0 9 ) * 2 / RTX 3 0 x x /4 0 x x series, o r one o f th e d ata c e n te r g ra p h ­
A
( 1 0 0 0 3 ) = 1 4 . 0 0 GB ics cards, Like th e A 10 0 o r H 100 series, are th e pre ferred
choices fo r ru n n in g LLMs. These to p o u t a t ~ l,0 0 8 G B /s
The re q uire d m em ory needed to run a m od el is h ig h e r (RTX 40 9 0 24G B), 2 ,0 3 9 G B /s (A 1 00 80GB) and 3 ,0 7 2 G B /s
s t ill. The ex ac t a m o u n t needed is , am ong o th e r th in g s , (H 10 0 80GB) res p ec tiv e ly .

codem ag.com You're M issing O u t o n O pe n-S o urce LLMs!

 It's im p o rta n t fo r a m o d el to f i t c o m p le te ly in to th e a v a il­ a rc h ite c tu re . The M l and M2 Pro have 205G B /s m em ory
able VRAM o f a g ra ph ic s card to ta k e ad va ntag e o f th e b a n d w id th (M3 154 G B /s) and can be e q u ip p ed w ith up to
a v a ila b le m em ory b a n d w id th and t h a t i t s n o t fo rc e d to be 32GB (M3 36GB) RAM, o f w h ic h ~24GB can be used fo r
s p lit betw e en th e gra ph ics card's and th e system 's m em o­ an LLM. ALL M Max series processors com e w ith 41 0G B /s
ry. W h ile po ss ib le , t h a t s p lit in c u rs a severe pe rfo rm ance b a n d w id th and up to 128GB RAM, and th e a v a ila b le M l
p e n a lty due to th e a fo re m e n tio n e d lim ita tio n s o f CPU and and M2 U ltra have 820 G B /s b a n d w id th w ith up to 192GB
system RAM. As y o u can see, a consu m er g ra ph ic s card is RAM. C onsidering t h a t m em ory is an expensive and lim it ­
hard pressed to run Larger m odels because i t has lim ite d in g fa c to r fo r GPUs, th is can be an a ttra c tiv e w ay o f e v a l­
VRAM. Y o u 'll have to b uy m u ltip le cards and d is trib u te th e u a tin g large m odels, s e rv in g LLMs fo r s m a ll team s , o r fo r
load to run la rg er m odels. a p p lic a tio n s t h a t d o n 't ex perie nce h ig h re q ue st vo lum es .

An u n lik e ly b u t c o m p a ra tiv e ly p ric e -e ffe c tiv e c o m p e ti­ I'm using a M acBook Pro w ith M l Pro and 32GB RAM fo r
t o r is th e A p p le M -series SoC w ith th e ir u n ifie d m em ory m y e xp e rim e n ts and co n sid e r i t u s e fu l fo r e v e ry th in g from

T he im a g e s h o w c a s e s a g o ld e n d e e r fig u rin e , w h ic h is th e m a in s u b je c t o f th e p h o to g ra p h . T h e d e e r s ta n d s on its h in d legs,

w ith a ga ze d ire c te d to th e le ft. Its b o d y is a d o rn e d w ith in tric a te p a tte rn s an d d e s ig n s in s h a d e s o f b lu e an d g o ld . T h e
b a c k g ro u n d fe a tu re s b lu rre d lig h ts , p ro v id in g a c o n tra s t th a t fu rth e r a c c e n tu a te s th e s ta tu e . T h e p o s itio n in g o f th e d e e r in
th e fo re g ro u n d , a lo n g w ith th e b lu rre d b a c k g ro u n d , c re a te s a sen se o f d e p th an d fo c u s o n th e fig u rin e its e lf. T h e re are no
d is c e rn ib le te x ts o r o th e r o b je c ts in th e im ag e. T he re la tiv e p o s itio n b e tw e e n th e d e e r a n d th e lig h ts s u g g e s ts th a t th e y are
lo c a te d at d iffe re n t d is ta n c e s fro m e a c h o th e r. T h e o v e ra ll c o m p o s itio n o f th e im a g e is b a la n c e d w ith th e g o ld e n d e e r s ta tu e
as its c e n tra l e le m e n t.

■ E x p o rt as S c re e n s h o t

P to s en d , s h ift ♦ fo r n e w lin e

M o d e l D o w n lo a d s

F ig u re 1 : U sing a v is io n m od el in LM S tu d io

28 You're M issing O u t o n O p en -S ou rce LLMs! codem ag.com

 to -T e x t, T ext-to-S peech, V ecto r DBs, S table D iffu s io n , and Common Models
m ore. And th e web U I makes i t easy to d ep lo y even on
An unlikely but comparatively
hosted m achines t h a t lack a desktop e n viro n m e n t. It has become somewhat
price-effective competitor is difficult to tell all the different
the Apple M-series SoC. W he the r y o u use lla m a .c p p d ire c tly , LM S tu d io , o r th e models apart from each
T e x t-g e n e ra tio n w eb U I, y o u 'll d ow n lo a d a m ode l, add i t other. As a reference point,
to a s p e c ifie d fo ld e r, and s e le c t i t fo r in fe re n c e . Y o u 'll you can take a look at the
have th e a b ilit y to c o n fig u re a system message o f y o u r list below. You'll often find a
1 6 - b it 7B m odels to 4 - b it q u a n tiz e d ve rsion s o f 33B pa­ ch oice and s e t in fe re n c e param eters lik e T em perature, combination of these in the
ra m e te r m odels. Max Tokens, TopP, TopK, and m any m ore, to fin e -tu n e th e
name of a model on Hugging
Face.
responses before p o s tin g y o u r q u e s tio n . The LLM w ill now
I f y o u 're lo o k in g a t e v a lu a tin g and ru n n in g large m odels, h a p p ily respond to y o u r q ue stio n s.
LLaMA 1 and 2
th e re 's an a lte rn a tiv e to th e large u p fro n t in v e s tm e n t in to
th e hardw are. Several h o s tin g services have cropped up Falcon
t h a t s e ll c o m p u tin g on an h o u rly basis (u s u a lly b ille d by C o nclu sio n
Alpaca
th e m in u te o r se con d). One o f th es e services is RunPod You have o n ly reached th e f ir s t peak on y o u r jo u rn e y in to
( h ttp s ://w w w .ru n p o d .io ) t h a t o ffe rs secure e n v iro n m e n ts th e Land o f o pe n-so urce LLMs. From here, y o u can see GPT4AII
to run w orklo a d s on h ig h -e n d da ta c e n te r GPUs lik e th e th e fa r-s tre tc h e d m o u n ta in ranges before y o u t h a t hold
Vicuna
H 100. These ty p e s o f services are n o t o n ly u s e fu l fo r endless p o s s ib ilitie s fo r e x p lo ra tio n and disco very. There
ru n n in g in fe re n c e on m odels, b u t also fo r fin e - tu n in g a are m any ways to go fro m here, w h e th e r y o u decide to d ig OpenBuddy
m od el to y o u r s p e c ific needs. in to research papers and learn m ore a b o u t th e d iffe re n t
Pygmalion
te c h n iq u e s used to im p ro v e LLMs, s ta r t b u ild in g an ap ­
p lic a tio n to im p ro v e an in te r n a l process, o r ex plore th e WizardLM
A re W e T here Yet? c a p a b ilitie s held by th e la rg e s t o f a v a ila b le ope n-so urce
Starcoder
You've Learned a b o u t th e m os t im p o rta n t aspects o f ru n ­ LLMs and th e cha lle ng es t h a t com e w ith ru n n in g th e m .
n in g o pe n-so urce LLMs a t th is p o in t, in c lu d in g w hy y o u 'd Mistral
use one, w h a t d iffe r e n t ty p e s th e re are, how to choose I f y o u are cu rio us a b o u t using LLMs in y o u r C# a p p lic a ­
StableLM
fro m th e th ou sa nd s o f m odels, and w h a t hardw are is re­ tio n s , lo o k o u t fo r a g u id e on using th is e x c itin g te c h n o l­
q uire d . o gy in one o f th e n e x t issues o f CODE M agazine! Deepseek

Mixtral MoE
Now t h a t w e're n e a rin g th e end o f o u r jo u rn e y to g e th e r, P h ilip p Bauer
one la s t q u e s tio n rem ains: W h a t to o ls can y o u use to run CODE Phi-2jihgfedcbaZYXWVUTSRQPONMLKJIHGF
an LLM on y o u r m achine?

L lam a.cp p is th e w orkh orse o f m any o pe n-so urce LLM

p ro je c ts ( h ttp s ://g ith u b .c o m /g g e r g a n o v /lla m a .c p p ).
It's a C/C++ Library t h a t prov ides s u p p o rt fo r LLaMA and
m any o th e r m od el a rc h ite c tu re s , even m u lti-m o d a l ( v i­
s io n ) m odels. I t made macOS a firs t-c la s s c itiz e n fo r LLMs,
a k in to W indow s and Lin u x t h a t s u p p o rt N V ID IA and AMD
GPUs. A s tro n g c o m m u n ity w orks tire le s s ly to im p ro v e i t .
S everal pro vid e d exam ples g iv e y o u a w ay to ex plore its
c a p a b ilitie s and a server prov ides an O penA I A P I c o m p a t­
ib le w eb A P I. T his is th e ch oice fo r y o u i f y o u w a n t to
im p le m e n t an LLM d ire c tly in to y o u r a p p lic a tio n , e ith e r
in C/C++ o r w ith one o f th e m any b in d in g s fo r Languages
fro m C# to Rust.

The m os t c o n v e n ie n t w ay to g e t s ta rte d is LM S tu d io fo r
W indow s and macOS ( h ttp s ://L m s tu d io .a i). This close d-
source, n o n -c o m m e rc ia l-u s e app a llo w s th e user to search
th e H ug ging Face database fo r a m od el and d o w n lo a d i t
d ire c tly w h ile h e lp in g w ith th e s e le c tio n o f th e c o rre c t
fo rm a t fo r y o u r p la tfo rm . I t prov ides an in t u it iv e in te rfa c e
and can expose an O penA I A P I-c o m p a tib le server y o u can
dev elop a p p lic a tio n s a g a in s t. It fe a tu re s m u lti-m o d a l
m od el s u p p o rt fo r v is io n m odels t h a t can analyze and
describe im ages t h a t are s e n t a lo n g w ith a te x t-p ro m p t.
U nder th e hood , i t uses th e lla m a .c p p package to pro vide
in fe re n c e (see F ig u re 1 ) .

For more advanced use-cases, th e O o b ab o o g a T ext g e n e ra ­

t io n w e b U I is a g re a t choice ( h ttp s ://g ith u b .c o m /o o b a -
b o o g a /te x t-g e n e ra tio n -w e b u i). I t can use m u ltip le m odel
back-ends beyond llam a.cp p, ad ding to its v e rs a tility . I t
sup po rts extensions ( b u ilt- in and c o m m u n ity ) fo r Speech-

codem ag.com You're M issing O u t o n O pe n-S o urce LLMs!

O N LIN E Q U IC K ID 2403051

Prototyping LangChain
Applications Visually Using Flowise
In my previous article in the July/August 2023 issue of CODE Magazine (https://www.codemag.com/Article/2307041/
An-lntroduction-to-OpenAI-Services), I gave you an introduction to OpenAI services. One of the topics I discussed was howto use
LangChain to build an LLM-based application. LangChain is a framework designed to simplify the creation of applications using

La rge L a n g u a g e M o d e ls . I t " c h a in s " t o g e t h e r v a r io u s c o m ­ • P ro v id e s a c o m m a n d lin e w h e re y o u c an i n s t a l l d i f ­

p o n e n ts t o c re a te c o m p e llin g A l a p p lic a tio n s t h a t can f e r e n t v e rs io n s w it h a s in g le c o m m a n d , s e t a d e ­
q u e ry v a s t a m o u n ts o f u p - t o - d a t e d a ta . f a u lt , s w itc h b e tw e e n th e m a n d m o re .

F or m acO S , ty p e t h e f o llo w in g c o m m a n d in T e rm in a l t o
in s ta ll n v m :
LangChain is a framework
$ curl -o-
designed to simplify the creation of https://raw.githubusercontent.com/
W e i-M e n g L e e applications using Large Language nvm-sh/nvm/v0.39.2 /in s ta ll. sh | bash
w e im e n g le e @ L e a rn 2 d e v e lo p .n e t Models. It "chains" together various
h t t p : / / w w w . Learn 2deveLop. n e t Y ou c an a ls o g e t t h e a b o v e c o m m a n d fro m h t t p s : / / g i t h u b .
@ w e im e n g L e e components to create compelling c o m / c r e a t io n ix / n v m , w h e re y o u ca n f in d t h e c o m m a n d t o

W ei-M eng Lee is a Al applications that can query vast i n s t a l l t h e la t e s t v e rs io n o f n v m .

te c h n o L o g is t and fo u n d e r amounts of up-to-date data. O nce t h e in s t a lla t io n is d o n e , t y p e t h e f o llo w in g com ­

o f D eveloper Learnin g
m a n d in T e rm in a l:
S o lu tio n s ( h ttp ://w w w .
Learn2deveLop.net),
a te c h n o lo g y co m p any $ nano - / . zshrc
s p e c ia liz in g in han ds-on To t h e n o v ic e , L a n g C h a in c an be q u it e o v e r w h e lm in g a n d
tr a in in g on th e Latest o v e r ly c o m p le x . A n d u n le s s y o u a re a d e v e lo p e r, L a n g ­ A p p e n d t h e f o llo w in g lin e s t o t h e .z s h r c f i l e a n d s a v e i t :
te c h n o lo g ie s . W ei-M eng C h a in re m a in s la r g e ly o u t o f re a c h t o m o s t p e o p le — u n t i l
has m any years o f tr a in in g F lo w is e ( h t t p s :/ / f l o w i s e a i.c o m ) . # put this in one line
expe rien ce and his t r a in ­ export NVM_DIR=
in g courses place s p e c ia l F lo w ise is a L o w -c o d e /n o co d e d ra g -a n d -d ro p t o o l t h a t m akes "$([ -z "${XDG_CONFIG_HOME-}" ] &&
em phasis on th e Learning- i t ea sy f o r p e o p le (p ro g ra m m e rs a n d n o n -p ro g ra m m e rs a lik e ) p rin tf %s "${HOME}/.nvm" || p rin tf %s
b y -d o in g ap pro ach . t o v is u a liz e a n d b u ild LLM ap ps . In s te a d o f w r itin g co d e us­ "${XDG_CONFIG_HOME}/nvm")"
His han ds-on approach in g th e LangC hain fra m e w o rk , y o u can j u s t d ra g -a n d -d ro p
to le a rn in g p ro g ra m m in g c o m p o n e n ts (k n o w n as no de s in F lo w is e ) a n d c o n n e c t th e m . # put this in one line
makes u n d e rs ta n d in g th e
I f in d i t v e ry u s e fu l t o g e t s ta rte d , an d as I e x p lo re deeper, [ -s "$NVM_DIR/nvm.sh" ] &&
s u b je c t m uch eas ier th a n
i t m akes m e a p p re c ia te LangC hain eve n m ore. \. "$NVM_DIR/nvm.sh"
re a d in g books, tu to ria ls ,
and d o c u m e n ta tio n .
I n t h is a r t ic le , T i l w a lk y o u t h r o u g h s o m e o f t h e k e y fe a ­ R e s ta rt T e rm in a l.
His nam e re g u la rly appears
tu r e s o f F lo w is e . I n p a r tic u la r , y o u 'll b u ild a n u m b e r o f
in o n lin e and p r in t p u b li­
fu n a p p s , in c lu d in g h o w t o b u ild c h a tb o ts t h a t w o rk s lik e F or W in d o w s , d o w n lo a d th e la t e s t n v m - s e tu p .e x e file
c a tio n s such as DevX.com ,
ChatG PT, an a p p t h a t q u e rie s y o u r o w n d a ta , a n d an a p p fr o m h t t p s : / / g it h u b . c o m / c o r e y b u t le r / n v m - w in d o w s /
M ob iF orge.com , and
t h a t 's a b le t o a n a ly z e y o u r CSV d a ta f ile s . W it h o u t f u r t h e r re le a s e s . T h e n , d o u b le - c lic k t h e n v m - s e tu p .e x e f i l e a n d
CODE M agazine.
d e la y , le t's g o ! i n s t a l l n v m in C :\n v m a n d n o d e js in C :\P ro g ra m F ile s \
n o d e js .

Installing Flowise Locally

T h e re a re a c o u p le o f w a y s t o g e t F lo w is e u p a n d r u n n in g .
L e t's g o th r o u g h t h e f i r s t m e th o d t o i n s t a l l F lo w is e on It's important that the installation
y o u r m a c h in e . As F lo w is e is b u i l t u s in g N o d e .js , y o u n e e d path for nvm have no spaces or else
t o f i r s t i n s t a l l N o d e .js .
you'll have problems using nvm
Prerequisites: Installing Node.js later on.
T he e a s ie s t w a y t o in s t a l l N o d e .js is t o i n s t a l l n v m (N o d e
V e rs io n M a n a g e r) f ir s t , n v m is a t o o l f o r m a n a g in g d i f f e r ­
e n t v e rs io n s o f N o d e .js . I t :
O nce n v m is in s t a lle d , y o u c a n i n s t a l l N o d e .js . To i n s t a l l
• H e lp s y o u m anage and s w itc h b e tw e e n d if f e r e n t th e L a te s t v e r s io n o f N o d e .js , u se t h e f o ll o w in g com ­
N o d e .js v e rs io n s w it h ease. m and:

Prototyping LangChain Applications Visually Using Flowise codem ag.com

$ nvm in s ta ll node $ mkdir flowise
# cd flowise jihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA
To use t h e la t e s t v e rs io n o f N o d e .js , use t h e f o llo w in g
com m and: C reate a f ile na m ed D o c k e r f ile a n d p o p u la te i t w ith th e c o n ­
t e n t , as s h o w n in L is t in g 1 .
$ nvm use node
The D o c k e rfile co ntain s th e in s tru c tio n s to b u ild a Docker image.
Installing Flowise
To in s t a l l F lo w is e , y o u ca n use n p m (N o d e P ackage M a n ­ N e xt, ty p e th e fo llo w in g c o m m a n d t o b u ild a D ocke r im a g e
a g e r ) , a t o o l t h a t c o m e s w it h N o d e .js . T yp e t h e f o llo w in g na m ed f lo w is e :
c o m m a n d in T e rm in a l t o i n s t a l l F lo w is e u s in g n p m :
$ docker build --no-cache - t flowise .
$ npm in s ta ll -g flowise
L is t in g 1 : C o n te n t o f D o c k e rfile
O nce t h e in s t a lla t io n is d o n e , y o u c an n o w s t a r t u p F lo ­
w is e u s in g t h e f o llo w in g c o m m a n d : FROM node: 18-alpine

USER root
$ npx flowise start
RUN apk add --no-cache g it
RUN apk add --no-cache python3 py3-pip make g++
Installing Flowise Using Docker # needed for pdfjs-dist
RUN apk add --no-cache build-base cairo-dev pango-dev
T h e s e c o n d m e th o d t o i n s t a l l F lo w is e is t o use D o cke r. For
# In s ta ll Chromium
t h is , I'm g o in g t o a s s u m e t h a t y o u a lre a d y h a v e D o c k e r
RUN apk add --no-cache chromium
in s ta lle d a n d t h a t y o u h a v e s o m e b a s ic k n o w le d g e o f i t .
I f y o u a re n e w t o D o ck er, r e fe r t o m y a r t ic le I n t r o d u c t i o n ENV PUPPETEER_SKIP_DOWNLOAD=true
ENV PUPPETEER_EXECUTABLE_PATH=/usr/bin/chromium-browser
to C o n ta in e r iz a tio n U s in g D o c k e r in th e M a r c h /A p r il
2021 is s u e o f CODE M a g a z in e ( h ttp s ://w w w .c o d e m a g . # You can in s ta ll a specific version like :
c o m / A r t i c l e / 2 1 0 3 0 6 1 / I n t r o d u c t i o n - t o - C o n t a in e r i z a t io n - # flowi se@l. 0.0
U s in g -D o c k e r). RUN npm in s ta ll -g flowise

WORKDIR /data
I n T e rm in a l ( o r C o m m a n d P r o m p t), c re a te a n e w d ir e c t o r y
a n d c h a n g e i n t o t h is n e w d ir e c to r y : CMD ["flowise"."start"]

lo c a lh o s t

F lo w is e A l

C h a t f lo w s

M a rk e tp la c e s
Chatflows oo
□ o Add N ew

A s s is t a n t s

a r e d e n t ia ls

V a r ia b le s

A P I Keys N o C h a t f lo w s Y e t

D is p la y a m e n u

F ig u r e 1 : F lo w is e u p a n d r u n n in g in t h e w e b b ro w s e r

codem ag.com Prototyping LangChain Applications Visually Using Flowise 31

 to citho it

U n title d c h a tflo w kjihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA B

Figure 2: You can sta rt build ing your LangChain application by adding nodes in to the em pty canvas o f your new Flowise project.

You can now use th e n ew ly b u ilt flo w is e Docker im ag e to • ♦ • H 1 w

C to c a ih o e t O + )

create and run a Docker c o n ta in e r (also nam ed flo w is e ;

as s p e c ifie d using th e —n a m e o p tio n ) : < U n title d c h a tflo w (3 ©

$ docker run -d --name flowise -p 3000:3000 flowise

The F low ise app in te r n a lly Listens on p o rt 3 00 0. The usage Add N odes

o f th e - p o p tio n in th e Docker com m and s ig n ifie s t h a t

th e Docker c o n ta in e r w ill be c o n fig u re d to lis te n on p o rt
30 00 e x te rn a lly (th e f ir s t 30 00 in 3 0 0 0 :3 0 0 0 ) and fo rw a rd
t h a t tr a f fic to p o rt 30 00 in te rn a lly , a lig n in g w ith th e p o rt
w here F low ise is a c tiv e ly lis te n in g . A g e n ts v

Cache v

L a u n c h in g F lo w is e C ham a v

Now t h a t Flowise is in s ta lle d and ru n n in g (e ith e r lo c a lly

C h a t M o d a l» v
using Node.js o r using D ocker), you can load Flowise using
a web browser. Type h ttp ://L o c a lh o s t:3 0 0 0 /in th e URL bar D o c u m e n t L o a d e rs v

and you sh ou ld see Flowise, as show n in F ig u re 1 .

E m b e d d in g s v

LLM * v

C re ating a S im ple L an gu ag e T ransla tor

M em o ry v
Click the Add New bu tton to create a new Flowise project. You
should now see the canvas fo r your new project (see Figure 2 ). M o d e r j t lo n v>

O u tp u t P a r s e r s v
Adding Nodes
To b u ild y o u r LLM -based a p p lic a tio n s , y o u add nodes to P ro m p t* v

th e p ro je c t. Nodes are th e b u ild in g block s o f y o u r Flowise

R e tr ie v e r s v
a p p lic a tio n .
T e x t S p litte r s v

T o o ls sr

The various nodes in Flowise map U tib tie s v

to corresponding components in V e c t o r S to r e s o te e tc a riM c v

LangChain. V e c t o r S to r e s v

+ - n a
D isolay a menu
To add a node to th e canvas, c lic k th e + b u tto n to d is ­
p la y a ll th e a v a ila b le nodes. ALL o f th e a v a ila b le nodes F ig u re 3 : Nodes are o rg an iz ed in to groups.

32 Prototyping LangChain Applications Visually Using Flowise codem ag.com

 OpenAI ©

In p u ts

o
• C ache ____

C onnect C redential *

- C re a te N e w -

g p t-3 .5 -tu rb o -in s tru c t

Figure 5: Configuring the OpenAI API Key in the OpenAI node

T em perature

0.7

|>-| Prompt Template

A d d itio n a l P a ram eters

In p u ts
O utput

OpenAI • L a n g c h a in H u b ■

T e m p la t e * **
F ig u re 4 : Use th e O penA I node to use a m od el from
O penA I.
T r a n s la te th e p ro v id e d { s e n te n c e }
fr o m E n g lis h to C h in e s e a s w e ll a s
are o rg an iz ed in to groups, such as A g e n ts , C ache,
Japanese
C h ain s , C h a t M o d e ls , e tc . (see F ig u re 3 ) . You can expand
each o f th es e groups to v ie w th e va rio u s nodes. A n s w e r:

For th is p ro je c t, Lets s ta r t o f f w ith a s tra ig h t-fo rw a rd ta sk.

Let's b u ild a language tra n s la to r t h a t tra n s la te s w h a te v e r
th e user ty p e s in in to Chinese as w e ll as Japanese. F o r m a t P r o m p t V a lu e s

F o r m a t P r o m p t V a lu e s

You can apply for an OpenAI API

key at: https://platform.openai.com/ O u tp u t
account/api-keys. Note that this is a
chargeable service. P r o m p t T e m p la t e •

The f ir s t node to add is th e O p e n A I node (lo c a te d u n ­ F ig u re 6 : C o nfigure th e P ro m p t T em plate node.

der th e LLMs g ro u p ). Drag and drop th e O p e n A I node
o n to th e canvas (see F ig u re 4 ) . You'Ll make use o f th e
g p t - 3 . 5 - t u r b o - in s t r u c t LLM pro vid e d by O penA I. N ext, y o u 're g o in g to add th e P r o m p t T e m p la te node
(lo c a te d u n d e r th e P ro m p ts g ro u p ). You'Ll c re a te th e
To make use o f th e LLM a t O penA I, y o u need to have an p ro m p t to in s t r u c t th e LLM to p e rfo rm th e tra n s la ­
O penA I A P I key. Take n o te t h a t y o u w ill be charged based tio n fro m E nglish to Chinese and Japanese. Type th e
on y o u r usage. fo llo w in g se nte nc es in t o th e T e m p la te te x tb o x (see
F ig u re 6 ) .
U nder th e C o n n e ct C r e d e n tia l s e c tio n o f th e O p e n A I
node, c lic k th e d ro p -d o w n b u tto n and s e le c t C re a te N ew . Translate the provided {sentence} from
Give y o u r c re d e n tia l a nam e and ty p e in y o u r O penA I A P I English to Chinese as well as Japanese.
key (see F ig u re 5 ) . Then c lic k A d d . Answer:

codem ag.com Prototyping LangChain Applications Visually Using Flowise 33

 To run th e p ro je c t, y o u f ir s t need to c lic k th e Save P ro j­
The th ir d and f in a l node y o u need to add is th e jihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA
LLM
C h ain node (Located un d er th e C h ain s g ro u p ). T his node e c t b u tto n to save th e p ro je c t. When p ro m p te d , name
takes in an LLM as w e ll as a p ro m p t te m p la te (as w e ll as th e p ro je c t M y T ra n s la to r. Then, c lic k th e C h a t b u tto n to
som e o th e r o p tio n a l nod es). C onn ect th e th re e nodes t h a t b rin g up th e c h a t w in d ow . F ig u re 9 shows th e response
yo u 'v e added, as show n in F ig u re 7 . re tu rn e d by th e O penA I LLM a fte r y o u ty p e d G ood m o r n ­
in g ! The response re tu rn e d is th e se nte nce tra n s la te d
Testing the Project in to Chinese and Japanese.
You're now ready to te s t th e p ro je c t. A t th e to p r ig h t co r­
ner o f th e page, th e re are se veral b u tto n s (see F ig u re 8 ) . Downloading the Project
Once th e p ro je c t is saved, y o u can d ow n lo a d a copy so
t h a t y o u can:

O penAI • Load i t back to Flow ise la te r on .

• P ro g ra m m a tic a lly c a ll y o u r Flow ise p ro je c t using

In p u ts
languages such as P ython o r Ja va S crip t.

• C a ch e
To d o w n lo a d th e Flow ise p ro je c t, c lic k on th e P ro j­
e c t S e ttin g s b u tto n and c lic k E x p o rt C h a tflo w (see
F ig u re 1 0 ) .
C o n n e c t C re d e n tia l *

O pen A I API Key ’

For th is p ro je c t, a JSON f ile nam ed M y T r a n s la to r C h a t-
flo w .js o n is do w n lo a de d o n to th e lo c a l co m p uter.

M odel N am e Using the Flowise Project Programmatically

W ith th e p ro je c t d o w n lo a de d, y o u can now make use o f
g p t-3 5 - tu r b o - in s tr u c t *
i t p ro g ra m m a tic a lly . Flow ise provides a nu m be r o f ways
to c a ll y o u r p ro je c t p ro g ra m m a tic a lly . C lick on th e b u tto n
la b e lle d < / > and y o u 'll see th e List o f o p tio n s show n in
T e m p e ra tu re
F ig u re 1 1 .
o .7 :
You can:
• Embed y o u r p ro je c t in a HTML w eb page.
A d d itio n a l P a ra m e te rs
2 2 LLM Cham • Embed y o u r p ro je c t in a React w eb a p p lic a tio n .
• Call y o u r p ro je c t as a w eb service using P ython o r
O u tp u t Ja va S crip t.
In p u ts

1
O penA I
M
• Language M odel •

♦ P ro m p t’

[> 7 | P r o m p t T e m p la te
• O u tp u t P a rs er

In p u ts
• In p u t M o d e ra tio n O

L a n g c h a in H u b />'.
C ham N a m e

T e m p la te ♦ •;

T ra n s la te th e p ro v id e d (s e n te n c e )
fro m E n g lis h to C h in e s e as w e ll as ■ O u tp u t

Japanese
A nsw er
L L M C h ain * •

<_______________________________________ /

F o rm a t P ro m p t V a lu e s

•
F o rm a t P ro m p t V a lu e s

O u tp u t

P fo m p tT e m p la te <r

F ig u re 8 : The va rio u s b u tto n s fo r c o n fig u rin g y o u r

F ig u re 7 : C onnect a ll th e nodes to g e th e r. a p p lic a tio n

34 Prototyping LangChain Applications Visually Using Flowise codem ag.com

 lo c a lh o s t

< My T ranslator </> ® ©

M oo? N am e

g p t -3 5 t u r b o - ln s t r u c t *

T e m p e r M u re

0 7 :

A d d itio n a l P a r a m e te r s G o o d m o r n in g 1
G D
UN

O u tp u t

C h in e s e ( Z S o s h a n g h A o 1) J a p a n e s e
O penA I •
• L an g , Z iiJ c L £ I ( O h a y o g o z a i m a s u 1)

< /> ® ©
e P’om

E ) P r o m p t T e m p la te
• O u tp l
F2 S ta r te r P r o m p ts
in p u ts
• In p u t
£3 V ev« b*es!>ag«*&
L a n g c h a m H ub Jy.
C h a ff
(Q D u p lic a te C h s tf lo w
T e m p la te •

7) L o a d C h a t f lo w
T r a n s la te th e p ro v id e d (s e n te n c e }

fr o m E n g lis h t o C h in e s e a s w e ll a s
Japanese E x p o rt C h a t f lo w
A nsw er

Q A n a ly s e C h a t f lo w

F o r m a t P r o m p t V a lu e s
Q D e le te C h a t flo w

F o r m a t P r o m p t V a lu e s

D is p la y a m a n u
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ Q v tx w t_ _ _ _ _ _ _ _ _ _ _
F ig u re 1 0 : E x p o rtin g y o u r
F ig u re 9 : T esting th e a p p lic a tio n F low ise a p p lic a tio n

lo c a lh o s t

F ig u re 1 1 : The va rio u s ways to use y o u r Flow ise a p p lic a tio n p ro g ra m m a tic a lly

codem ag.com Prototyping LangChain Applications Visually Using Flowise 35

 lo c a lh o s t jihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA

F ig u re 1 2 : S ha rin g th e c h a tb o t w ith o th e r users

• Call y o u r p ro je c t as a w eb service using th e CURL import requests

u t i li t y on th e com m and Line.
• Share y o u r p ro je c t as a c h a tb o t w eb a p p lic a tio n . API_URL = "http://localhost:3000/api/" +
"vl/predi Ction/a46b75ab-89e5"+
Let's ta k e a Look a t som e exam ples. For accessing th e " -446d-8323-847d83094791"
p ro je c t as a w eb service us ing P yth o n, y o u can use th e
fo llo w in g code s n ip p e t: def query(payload):
response = requests.post(API_URL,
j son=payload)
return response.json()

output = query({
"question": "Hey, how are you?",
})
print(output)

Hi th e re ! H ow can I help?
The re s u lt re tu rn e d w ill lo o k Like th is (fo rm a tte d fo r c la r-
it y ) :
H ow d o I g o to th e ne a re st tra in sta tio n ?
{
'te x t 1 :
C hinese:
'\nChi nese: 1 , M l ?
J a p a n e s e :® ^ U © IR S T t ' I '7 ? "S’ Z)'? \nJapanese:
}

For accessing th e p ro je c t on th e com m and Line, y o u can

use th e CURL u t ilit y :

$ curl h ttp ://lo ca lh o st:3000/apiZvl/

ly p e y o u r q u e s tio n
prediction/a46b75ab-89e5-446d-8323-
847d83094791 \
_. P o w e r e d b y F lo w is e -X POST \
D is p la y a m e n u *
-d '{"question": "Hey, how are you?"}' \
F ig u re 1 3 : R un ning th e c h a tb o t as an in d e p e n d e n t w eb a p p lic a tio n -H "Content-Type: application/json"

36 Prototyping LangChain Applications Visually Using Flowise codem ag.com

 C h a tO p e n A I ®

©
In p u ts

• Cache Hi th e re ! H o w can I help?

C on nect C red en tial *

O penA IA PIKey w W here is S in g a p o re lo c a te d ’’

M od el N am e

g p t-3 5 -tu rb o w S in g a p o re is lo c a te d in S o u th e a s t A sia , at th e

s o u th e rn tip o f the M a la y P e n in s u la It is

T em perature s itu a te d ju s t n o rth o f th e e q u a to r an d is a c ity -

s ta te th a t c o n s is ts o f o n e m a in is la n d and
0 9 ;
sev eral s m a lle r is la n d s The c o u n try Is

b o rd e re d by M a la y s ia to th e n o rth and
A d d itio n a l P ara m ete rs C o n v e rs a tio n C h ain
CONY In d o n e s ia to th e s o u th The m a in is la n d o f

In p uts S in g a p o re is c o n n e c te d to M a la y s ia by a
O u tp ut
cau sew ay, and th e re are a ls o b n d g e s lin k in g it
C hatO penA I • « • Language M o d e l *
to v a rio u s s m a lle r is la n d s .

• M e m o ry *
B u ffe r M e m o r y

• D o cu m en t O D oes it have fo u r sea so ns ?

In p u ts

A d d itio n a l P a ra m e te rs
M e m o ry Key *

c h a t_ h is to ry O utp ut No, S in ga po re d o e s n o t have fo u r d is tin c t

s e a s o n s like m a n y o th e r c o u n trie s . Inste ad , it

C o n v e rs a tio n c h a in •
Input Key * e x p e n e n c e s a tro p ic a l ra in fo re s t c lim a te

c h a ra c te riz e d by u n ifo rm ly hig h te m p e ra tu re s ,

in p u t

O utput

B u ffe rM e m o ry w

F ig u re 1 5 : You can have a c o n v e rs a tio n w ith th e

F ig u re 1 4 : C o n n ec tin g th e nodes to g e th e r c h a tb o t and i t rem em bers th e c o n te x t o f th e c h a t.

The re s u lt re tu rn e d w ill Look Like th is : C rea tin g a C onv ers ationa l C h a tb o t

The second p ro je c t to create is a c o n v e rs a tio n a l c h a tb o t.
{"text":"\nChinese: < W f i ? U sing Flow ise, y o u can create a c h a tb o t t h a t fu n c tio n s
\nJapanese: fe7cM ?Tb'?"} v e ry m uch Like ChatGPT. Here are th e nodes t h a t y o u 'll be
using fo r th is p ro je c t:
You can also share th e c h a tb o t w ith o th e r users. To do
th a t , c lic k on th e S h a re C h a tb o t ta b and copy th e URL • C h a tO p e n A I: You use th is node to s p e c ify th e m odel
(see F ig u re 1 2 ) . y o u w a n t to use from O p enA I. A lso , y o u need to
s p e c ify y o u r O penA I A P I key, so ta k e n o te t h a t you
When y o u paste th e co pied URL in to a w eb brow ser, y o u 'll w ill be charged based on y o u r usage. For th is ex­
be p ro m p te d to e n te r a usernam e and passw ord. By de­ am ple, y o u 'll use th e g p t - 3 . 5 - t u r b o m o d e l.
fa u lt, Flow ise is s ta rte d w ith o u t a u th o riz a tio n p ro te c tio n . • B u ffe r M e m o ry : The m em ory to rem em ber y o u r c o n ­
You can e n te r a n y th in g fo r usernam e and passw ord and v e rs a tio n .
y o u 'll be a llo w e d accessed to th e c h a tb o t. I f y o u w a n t to • C o n v e rsa tio n C h ain : A co n v e rs a tio n ch a in th a t
bypass th e p ro m p tin g fo r usernam e and passw ord, check takes in a LLM and m em ory w ith th e p ro m p t te m ­
th e M a k e P u b lic o p tio n , as show n in F ig u re 1 2 . p la te alrea dy c o n fig u re d fo r c h a ttin g .

You can now see th e c h a tb o t. T his prov ides a c o n v e n ie n t F ig u re 1 4 shows how th e nodes are c o n n e c te d . N ote t h a t
w ay fo r users to make use o f y o u r p ro je c t s tra ig h t away fo r th is exam ple, th e in fe re n c in g (ru n n in g o f th e LLM) is
(see F ig u re 1 3 ) . p erfo rm ed by O penA I.

codem ag.com Prototyping LangChain Applications Visually Using Flowise 37

 Save th e p ro je ct and th en click th e jihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA
C hat b u tto n . You can cha t • H u g g in g F a c e In fe r e n c e : Use th is node to make use
w ith th e OpenAI LLM and fo llo w up w ith questions (see Fig­ o f a LLM ho ste d by H uggingFace. You need to spec­
ure 1 5 ) . As th e conversation chain is connected to th e b u ffe r if y y o u r H uggingFace Token key.
memory, you can m ain tain a conversation w ith th e LLM. • P ro m p t T e m p la te : C onfigures th e p ro m p t te m p la te .
• LLM C h ain : C onnects to th e LLM and p ro m p t te m ­
A lte rn a tiv e ly , i f y o u w a n t to b u ild a c h a t a p p lic a tio n p la te .
w ith o u t p a yin g fo r th e LLM (as in th e case o f O p e n A I),
y o u can use th e H u g g in g F a c e In fe r e n c e node. F ig u re 1 6 N ote t h a t fo r th e H u g g in g F a ce In f e r e n c e node, th e re are
show s how y o u can do t h a t using th e fo llo w in g nodes: tw o ways to use th e m odel:

** H u g g in g F a c e In fe re n c e The m ode! m is tfa la i/M ix tra l-8 x 7 B -v O .l is to o

large to be loaded a u to m a tic a lly (93GB >

In p u ts
10GB) Please use Spaces (

• Cache
h ttp s //h u q qtn q fa ce c o /s p a c e s

) or Inference E nd points (

C o n n e d C r e d e n tia l * h ttp s //h u g g in q face c o /in fe re n c e -

e nd po in ts
H u g g in g F a c e A P I

).

M odel O
F ig u re 1 7 : The e rro r d is p la ye d w hen th e m od el is to o
tn u M /fa lc o n -7 b in s tr u c t large fo r y o u r m achine

E n d p o in t 0

LLR
LLM Cham

A d d itio n a l P a r a m e te r s
in p u ts
d> H i t h e r e 1 H o w c a n I h e lp ?

O u tp u t • Language M odel *
T e ll m e a jo k e

H u g g ln g F a c e in fe r e n c e
• P ro m p t *

[> ,] P ro m p t T e m p la te
• O u tp u t P a rs e r
<V> " W h y d id t h e t o m a t o tu r n r e d 7 B e c a u s e it s a w

th e s a la d d r e s s in g '

In p u ts C h a in N a m e

L a n g c h a in H u b

O u tp u t
T e m p la te •

Q u e s tio n I q u e s tio n )
L L M C h a in
Answ er

F o r m a t P r o m p t V a lu e s

F o rm a t P r o m p t V a lu e s

O u tp u t

P ro m p t T em p i a te e

F ig u re 1 6 : Using th e H uggingFace In fe re n c e node fo r c h a ttin g F ig u re 1 8 : C h a ttin g w ith th e m od el from H uggingFace

38 * Prototyping LangChain Applications Visually Using Flowise codem ag.com

Copilot • Al • GitHub • .NET • Visual Studio • C# • Azure Al • Blazor • ChatGPT • ASRNET • OpenAI • and more...kjihgfed

DEVintersection
Conference

Microsoft Azure
+AI Conference
P O W E R E D BY

M icroso ft & Tech Intersection

M A Y 7 -9 WORKSHOPS MAY 5 ,6 & 10

MGM GRAND, LAS VEGAS, NEVADA
Let the Experts Guide You on Your A l Journey:

SCOTT H A N S E L M A N SCOTT H U N T E R KATHELEEN D A N W A H L IN M A D D Y M O N T A Q U IL A

Vice President, Vice President, Director DO LLAR D Principal Cloud Senior Program
Developer Community, Product Management Principal Program Developer Advocate, Manager, .NET MAUI,
Microsoft for Azure Developer Manager, Microsoft Microsoft Microsoft
Experience, Microsoft

JEFF FRITZ LESLIE R IC H A R D S O N M A R K U S EGGER Z O IN E R T E J A D A M IC H E LE LERO UX

Principal Program Program Manager, President and Chief CEO and Architect, BUSTAM ANTE

Manager, Microsoft Microsoft Software Architect, Solliance President & Architect,

CODE Group Solliance

REGISl
S D E V in te rs e ctio n .c o m X @ D E V in te rs e c tio n S A z u re A IC o n f.c o m X @ A z u re A IC o n f
8 in fo @ D E V in te rs e c tio n .c o m Q in fo @ A z u re A IC o n f.c o m

BONUS: Al for Decision Makers Track

Questions Answered, Strategies Delivered, Relatic

 • I f y o u s p e c ify th e m o d e l nam e, jihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA
tiiu a e /fa lc o n -7 b -in -
s t r u c t in t h is e xa m p le , th e m o d e l w i l l be d o w n lo a d e d to
Vector embedding, also known
y o u r c o m p u te r an d ru n Locally. I f y o u t r y t o use a m o d e l
th a t's to o Large, such as m i s t r a l a i / M i x t r a l - 8 x 7 B - v O . l , as word embedding or vector
y o u m ay g e t an erro r, as s h o w n in F ig u re 1 7 . representation, is a technique
• To use a Larger LLM (s u c h as m i s t r a l a i / M i x t r a l -
8 x 7 B -v O .l), yo u need to use H u g g in g F a c e I n f e r -
used in natural language processing
(NLP) and machine learning
to represent words or phrases
as numerical vectors.
pd7 P d fF H e
C h a r a c t e r T e x t S p litte r The idea behind vector embedding
In p u ts
In p u ts
is to capture the semantic
o T e x t S p litt e r relationships and contextual
C h u n k S iz e
information of words in a
P d f F ile •
1000 :

Invoice? p d f
continuous vector space.
C h u n k O v e r la p t , U p lo a d F ile
t

200 :
e n c e E n d p o in ts ( h t t p s : / / h u g g in g f a c e . c o / in f e r e n c e -
Usage *
e n d p o in t s ) , w h ic h ru n s t h e m o d e l o n t h e c lo u d ( b y
C u s to m S e p a ra to r O O ne docu m en t per p age H u g g in g F a c e ).

A d d itio n a l P a r a m e t e r s
F ig u r e 1 8 s h o w s t h e c h a t b o t u s in g t h e t i i u a e / f a l c o n - 7 b -
i n s t r u c t m o d e l.
O u tp u t
O u tp u t

C h a r a c t e r T e x tS p lit te r •
D ocum ent •
Querying Local Documents
C h a ttin g w ith an LLM is fu n , b u t in th e re a l w o rld , b u s in e s s ­
es are m ore in te re s te d in w h e th e r th e y are a b le t o m ake use
o f A l t o q u e ry t h e ir o w n d a ta . W ell, u s in g a te c h n iq u e k n o w n
F ig u re 2 0 : C o n n e c tin g th e C h a ra c te r T ex t S p litte r n o d e t o th e P d f F ile nod e as v e c t o r e m b e d d in g , y o u 're n o w a b le t o do ju s t t h a t .

H uggingFace Inference
Em beddings

In p u ts
THA NK YOU 1 L ozo do

S a t i t h e d ? R a le u s!
F m a la r € < M m w ith C o n n e c t C re d e n tia l *
• v e r y fta v w w vOu » u 6 m i|

S ta y U p d a te d !
S in g a p o r e F o lo w Ota tlo r a fo r m a .
• i t l r g p r W r A v o u C h a rV H u g g in g F a c e A P I
E n d le s s P a s s io n

240 M ACPHERSO N ROAD *0 5 -0 1 R e a c h o u t to m w e C h a t,

le t s se e h o w w e c a n h« f t

S t a te m e n t

M odel O

E n d p o in t O
O rd e r N o J f n O Q IX X M X X K X

P a y m e n t M e th o d M IX E D C A R D

D a te : 1 6 0 4 2 0 2 3
Your ordered items: statem ent No v x u e e a x x o o u i

N um ber U n it T o ta l O u tp u t
S h o p SKU S e lle r S K U P ro d u c t n am e
o f ite m s P rice P rice

R a z e r B la c k W id o w v 3 M e c h a n ic a l G a m in g K e y b o a r d • C h
1327170305 R a z e r B la c k w id o w
r o m a R G B L ig h tin g - C o m p a c t F o rm F a c to r - P r o g r a m m a
H u g g in g F a c e ln fe re n c e E m b e d d in g s <>
1 .S G A M Z 82 V 3 G a m in g K e y b o 1 1699 1699
b le M a c r o F u n c tio n a lity - U S B P a s s th r o u g h C o lo r.B la c k -
84533445 a r d • Y e llo w S w itc h
Y e llo w S w itc h (L in e a r)

F ig u r e 2 2 : C o n fig u r in g t h e H u g g in g F a c e In fe r e n c e
F ig u r e 2 1 : T he s a m p le in v o ic e PDF f i l e E m b e d d in g s n o d e

40 Prototyping LangChain Applications Visually Using Flowise codem ag.com

 P d < F e
£ £ "

In p u t s

• T e x t S p i t le r

P d f F ile •

Invoice? pdf

One d o cu m en t per page

(fg In -M e m o r y V e c to r S to re

In p u ts

O .it p u t

e D ocum ent

a E m b e d d in g s *

H u g g in g F a c e In fe r e n c e Top K O

E m b e d d in g s

In p u ts

O u tp u t
C o n n e c t C re d e n tia l •

H u g g in g F a c e A P I
M e m o ry R c tn c v e r

(Cq In -M e m o r y V e c to r S to re

E n d p o in t • In p u t s

• Docum ent

O u tp u t
• E m b e d d in g s *

H u r j g n n T -ic H n f ^ r del ir- q a •

TopK O

F ig u r e 2 3 : C o n n e c tin g a l l t h e n o d e s t o g e t h e r
O u tp u t

C o n v e r s a tio n a l R e tr ie v a l Q A

In t h is s e c tio n , y o u 'll b u ild an a p p lic a tio n t h a t a llo w s M e m o r y R e tr ie v e r

C h a in

yo u to q u e ry y o u r o w n PDF d o c u m e n t. Y ou'Ll n e e d t h e In p u ts

f o llo w in g n o d e s :
• Language M odel *

O p en A I
• C h a ra c te r T e x t S p litte r : Use t h i s node to s p lit a
• V e c t o r S te r e R e t n e v e r •
lo n g d o c u m e n t i n t o s m a lle r c h u n k s t h a t c an f i t i n t o In p u ts

y o u r m o d e l's c o n t e x t w in d o w .
• M e m o ry O
C ache
• P D F F ile : Lo ad s a PDF d o c u m e n t f o r p ro c e s s in g .
• H u g g in g F a c e In te r fa c e E m b e d d in g s : Use t h is R e tu r n S o u r c e D o c u m e n t s
C o n n e c t C r e d e n t al *
node to p e rfo rm e m b e d d in g . E m b e d d in g re fe rs t o
B
t h e r e p r e s e n ta tio n o f w o rd s o r s e n te n c e s as v e c to r s O p e n A I A P I Key

in a h ig h - d im e n s io n a l s p a c e . I t 's a w a y t o re p re ­
A d d it io n a l P a r a m e t e r s
s e n t w o rd s a n d s e n te n c e s in a n u m e r ic a l m a n n e r. M odel N am e

N o te t h a t in t h is e x a m p le , y o u c an a ls o m a k e use o f g p t - 3 .S - t u r b o - r n s t r u c t O u tp u t

t h e O p e n A I E m b e d d in g s n o d e , b u t t h i s w i l l in c u r
C o n v c r s o tio n o IR c tr ie v a lQ A C h a in •
c h a rg e s . I n c o n tr a s t, t h e H u g g in g F a c e In te r fa c e
T e m p e ra tu re
E m b e d d in g s n o d e uses t h e e m b e d d in g m o d e l fro m
o .7 :
H u g g in g Face, w h ic h is fre e .
• In - M e m o r y V e c to r S to re : Use t h is node to s to r e
A d d it io n a l P a r a m e t e r s
e m b e d d in g s in - m e m o r y a n d i t p e rfo rm s an e x a c t,
lin e a r s e a rc h f o r t h e m o s t s im ila r e m b e d d in g s .
O u tp u t
• O p e n A I: Use t h is n o d e t o m a k e use o f an LLM fro m
O p e n A I t o p e rfo rm q u e r y in g o f y o u r lo c a l d a ta . O penAI •

• C o n v e r s a t io n a l R e t r i e v a l Q A C h a in : Use t h i s n o d e
t o c re a te a r e tr ie v a l- b a s e d q u e s tio n a n s w e rin g c h a in
t h a t is d e s ig n e d t o h a n d le c o n v e r s a tio n a l c o n te x t. F ig u r e 2 4 : A d d in g t h e O p e n A I a n d C o n v e r s a tio n a l R e tr ie v a l QA C h a in n o d e

codem ag.com Prototyping LangChain Applications Visually Using Flowise 41

 /B C h a r a c te r T e x t S p lit te r (jjg In - M e m o r y V e c t o r S to re

in p u t* In p u ts

Chunk S u e Docum ent

1000 :
E m b e d d in g s •

C h u n k O v erlap Top K •

200 -
CO C o n v e r s a tio n a l R e tr ie v a l Q A
0 4
U sage * C h a in
C u s to m S e p a r a to r O O u tp u t
In p u ts
O ne d o c u m e n t per p ag e

M e m o r y R e trie v e r • Languag e M od el *

O u tp u t

O u tp u t
C b a ra c te rT e x tS p litte r • 3^) OpenAI

In p u ts

»- H u g g in g F a c e In fe r e n c e C ache

E m b e d d in g s
C o n n e c t C re d e n tia l *
In p u ts
A d d itio n a l P a r a m e te r s
O p e n A I A P I K ey v
C o n n e c t C re d e n tia l *
O u tp u t
H u g g in g F a c c A P I •*
M odel N am e
C o riv w rs flt'o n a lR *< ri€ v s lQ A C h d in •
g p t- 3 5 - tu r b o - m s t r u c t

T e m p e r e tu ie

0 .7 C

A d d itio n a l P a r a m e te r s

O u tp u t O u tp u t

H u g g m g F a c e tn fe r e n c e E m b e d d in g s * O p enA I •

The c o m p le te p ro je c t
F ig u re 2 5 : kjihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA

lo c a lh o s t

F ig u re 2 6 : C lic kin g th e
U psert b u tto n

F ig u re 2 7 : The steps pe rfo rm ed by th e U psert b u tto n

42 Prototyping LangChain Applications Visually Using Flowise codem ag.com

 < > lo c a lh o s t 0 © + Q

F ig u re 2 8 : You can now ask q u e stio n s p e rta in in g to y o u r PDF d o c u m e n t.

You can obtain a Hugging

Face API Token from https://
huggingface.co/settings/tokens.
C h a tO p e n A I

Inp uts
YV
C
C SV A g e n t
You'Ll learn how to add each o f th e above nodes and c o n ­
Cache
n e c t th e m in th e fo llo w in g steps.
In p u ts

F irst, add th e C h a ra c te r T e x t S p li t t e r node and c o n fig u re C o n ne ct C redential *

• Language M o d e l *
i t as show n in F ig u re 1 9 .
O penAI API Key ▼

T hen, add th e P d f F ile node to th e canvas as show n in C sv File • CBA

F ig u re 2 0 . To up lo ad a lo c a l PDF d o c u m e n t fo r q u e ry in g , T it a n iC - t r a in c s v
M od el N am e
c lic k th e U p lo a d F ile b u tto n and s e le c t th e PDF d o c u m e n t
t h a t y o u w a n t to use. T hen, c o n n e c t th e C h a ra c te r T e x t g p t-3 .5 -tu rb o w T, U pload File

S p li t t e r node to th e P d f F ile node.

A d d itio n a l P ara m ete rs

F ig u re 2 1 show s th e c o n te n t o f th e PDF d o c u m e n t t h a t T em perature

c o n ta in s th e o n lin e purchase o f an ite m .

o.9 : O u tp ut

N ext, add th e H u g g in g F a c e In t e r f a c e E m b e d d in g s node

A ge n tE xe cu to r *
to th e canvas and e n te r y o u r H ug ging Face A P I to k e n (see A d d itio n a l P ara m ete rs
F ig u re 2 2 ) .

O utput
The n e x t node to add is In - M e m o r y V e c to r S to re . Add
th is node to th e canvas and c o n n e c t i t to th e P d f F ile and C hatO penA I
H u g g in g F a c e In f e r e n c e E m b e d d in g s nodes as show n in
F ig u re 2 3 .

N ext, drag and drop th e O p e n A I node and s e t y o u r Ope­

n A I A P I key. F inally, add th e C o n v e rs a tio n a l R e tr ie v a l
QA C h ain node and c o n n e c t i t to th e o th e r nodes as
show n in F ig u re 2 4 . F ig u re 2 9 : U sing th e C hatO penA I and CSV A g e n t nodes

codem ag.com Prototyping LangChain Applications Visually Using Flowise 43

 lo c a lh o s t

F ig u re 3 0 : A skin g q u e stio n s based on th e c o n te n t o f th e CSV file

The c o m p le te p ro je c t is show n in F ig u re 2 5 . to ry answer. To use LLMs fo r a n a ly tic a l tasks, a com m on

approach in v o lv e s e m p lo y in g th e LLM to ge nerate th e
necessary code fo r th e query and s u b s e q u e n tly e x e c u tin g
The term upsert refers to an th e code in d e p e n d e n tly .

I
operation that inserts rows into
a database table if they don't already
In short, an agent helps you
exist, or updates them if they do.
accomplish your tasks without you
needing to worry about the details.

Before y o u can run th e p ro je c t, y o u need to c lic k th e

U p s e rt b u tto n , as show n in F ig u re 2 6 .
In LangC hain, th e re is a fe a tu re know n as a g e n ts . An
You'Ll see th e p o p u p , as show n in F ig u re 2 7 . a g e n t is a system t h a t decides w h a t a c tio n is to be tak en
by th e LLM, and i t trie s to solve th e pro blem u n t il i t
C lic king th e U p s e r t b u tto n perfo rm s a fe w o p e ra tio n s : reaches th e c o rre c t answer. A ge nts are designed to p er­
t e x t s p litt in g on th e PDF d o c u m e n t, c re a tin g em be dd ing s fo rm w e ll-d e fin e d tasks, such as ans w e rin g q ue stio n s,
using th e H uggingFace In fe re n c e Em beddings, and th e n g e n e ra tin g te x t, tra n s la tin g languages, su m m a rizin g te x t,
s to rin g th e v e c to rs in -m e m o ry . e tc .

Once th e u p s e rt is done, y o u can now c lic k on th e C h at For th is s e c tio n , I w a n t to show how y o u can make use o f
b u tto n to s ta rt th e c h a tb o t (see F ig u re 2 8 ) . th e CSV A g e n t in LangChain (and in F low ise) to perform
a n a ly tic a l tasks on a CSV file . The CSV A g e n t node o p e r­
ates by re a din g a CSV f ile in th e ba ck grou nd . I t em ploys
Using A g e n ts fo r A n a ly zin g Data th e Pandas DataFrame Library and uses th e P ython Lan­
LLMs are designed p rim a rily fo r g e n e ra tin g responses re­ guage to exe cute P ython query code gen erated by an LLM.
la te d to n a tu ra l Language u n d e rs ta n d in g . N evertheless,
th e y e x h ib it n o ta b le L im ita tio n s w hen faced w ith ana­ For th e CSV file , I 'l l be using th e T ita n ic tra in in g dataset
ly tic a l in q u irie s . For exam ple, w hen presente d w ith a CSV (Source o f Data: h ttp s ://w w w .k a g g le .c o m /d a ta s e ts /te d llh /
file and asked fo r a sum m ary o f its c o n te n ts , LLMs o fte n tita n ic -tra in . Licensing — Database Contents License (DbCL)
d e m o n s tra te Lim ited c a p a b ilitie s in p ro v id in g a s a tis fa c ­ v l.O h ttp s ://o p e n d a ta c o m m o n s .o rg /L ic e n s e s /d b c L /l-0 /).

44 Prototyping LangChain Applications Visually Using Flowise codem ag.com

 lo c a lh o s t © ± + Ci jihgfedcbaZY

F lo w is e A l

M a r k e tp la c e

J L C h a tflo w s

POPULAR

F lo w is e D o cs Q nA A P I A g e n t O p en A I API A ge n t A n to n y m
H o w is e D o c * G ith ub Q nA using Use O p e rA I f-u n c t» n A ge nt and G iven API docs, j g j n t O utpu t a n to n y m o f g iven user
c o n v e is a tio n a l re trie v a l QA c h a in C hain to a u to m a tic a lly decide a u to m a tic a l!/ d ecide wtwch A PI to in p u t u sin g fe w s h o t p ro m p t
w in ch API to cad. g en e ta im g url c a ll y e n c a t o y u il and body te m p la te bu ilt w ith e x a m p le s

"/. lie S ® and b o d y re q u e s t fro m rep ue sl fro m c o n v e rs a tio n

c o n v e is a tio n 0 s $ s
S? GO © & #
€7 CBA
& <x> ® ©
X

A u to G P T B abyA G I CSV A g e n t C ha t w ith a P o d c a s t

Use AutoGPT ■A u to n o m o u s a g e rt Use H atiyAG l to create ta sks and A n a ly s e a n d s u m m a r iz e C S V d a ta Engage w ith data so urce s such as
w ith cham o f th o u g h ts fo r s e lf re p rio n tize for a g rr e r o bjec tiv e YouTube Transcripts, G oogle, and
guided ta s k c o m p le tio n m ore th ro u g h in te llig e n t Q&A
in te ra c tio n s

ft & (?) Hi ®
® G ® ®
a

[P O P U L A fl]

C h a tG P T P Iu g in C la u d e LLM C o n v e rs a tio n a l C o n v e rs a tio n a l

Use ChatG PT P lu g in s w ith in Use A n th io p ic C laude w ith 2 0 tk Agent R e trie va l A g e n t
LangC hast a b s tra c tio n s w ith GET co n te x t w in d o w to ing es t w h o le
and POST T o o l* d o c u m e n t fo r QnA A c o n v e rs a tio n a l agent fo r a chat A ge nt o p tim iz e d fo r v e cto r
m o d e l w h ic h u tiliz e c h a t s p e c ific retrie va l d u rin g co n v e rs a tio n and
p ro m p ts a ns w erin g q u e s tio n s based o n
prev iou s dialogue

0 © B ® X
> X § © £

F ig u re 3 1 : Go to th e M arketplace fo r te m p la te s o f p o p u la r a p p lic a tio n s y o u can b u ild .

Let's create a new Flow ise p ro je c t and add th e fo llo w in g Chain p ro gra m m in g can be in tim id a tin g , Flow ise takes
c o m p on e nts to th e canvas: aw ay th e fe a r and prov ides a lo w -c o d e /n o -c o d e e x p e ri­
ence fo r y o u to g e t s ta rte d w ith A l p ro g ra m m in g . One
• C h a tO p e n A I: Rem em ber to e n te r y o u r O penA I A P I m ore th in g : I f y o u w a n t to Learn m ore a b o u t th e capa­
key. b ilitie s o f Flow ise, th e fa s te s t w ay is to learn fro m th e
• CSV A g e n t: C lick on th e U p lo a d F ile b u tto n to se­ te m p la te s p ro v id e d . F ig u re 3 1 shows th e M a rk e tp la c e s
le c t th e T ita n ic CSV file T ita n ic _ tr a in .c s v . in Flow ise w here y o u can g e t th e te m p la te s fo r p o p u la r
a p p lic a tio n s t h a t y o u can b u ild w ith Flow ise. Good Luck
F ig u re 2 9 show s how th e nodes are co n n e cte d . and have fu n !

Save th e p ro je c t and th e n c lic k th e C h a t b u tto n to d is p la y W ei-M eng Lee

th e c h a tb o t (see F ig u re 3 0 ) . You can now ask a n a ly tic a l CODE
q u e stio n s p e rta in in g to th e CSV file , such as:

• How m any m ale passengers were the re ?

• How m any passengers b e lo w th e age o f fiv e died?
• How m any passengers were m ore th a n 50 years old?

S u m m a ry
I hope th is a rtic le prov ides m o tiv a tio n fo r y o u to g e t
s ta rte d w ith LangChain p ro g ra m m in g . A lth o u g h Lang­

codem ag.com Prototyping LangChain Applications Visually Using Flowise

O N LIN E Q U IC K ID 2403061

Semantic Kernel 101: Part 2

In the previous installment (https://codemag.com/Article/2401091/Semantic-Kernel-101), I Introduced Semantic Kernel (SK),
Microsoft's framework for working with LLMs, which was in preview at the time. I didn't include any code in that article because
Semantic Kernel was still changing fast and almost every line of code would likely have been out of date before you read it.jihgfedc

So I p ro m is e d a f o llo w - u p a r t ic le t h a t e x p lo re d t h e c o d e . M o d ify y o u r P ro g ra m .e s f i l e t o lo o k lik e t h is :

S in c e t h e n , SK w e n t i n t o b e ta a n d th e n R eleas e C a n d i­
d a te , a n d , as I w r it e t h is , V I has j u s t re le a s e d a n d I e x ­ in te rn a l class Program
p e c t t h a t nam espaces, ty p e n a m e s , a n d a rg u m e n ts are {
n o w in t h e i r f i n a l fo r m , o r v e r y c lo s e t o i t . private s ta tic s trin g _endpoint;
private s ta tic s trin g apiKey;
To f o llo w a lo n g w it h t h e c o d e in t h e a r t ic le , y o u 'll n e e d
ac ce ss t o an LLM . I'm u s in g A z u re O p e n A I. As an a lte r n a ­ s ta tic async Task M ain(string[] args)
t iv e , y o u c an c o n n e c t t o an O p e n A I a c c o u n t. {
M ik e Y e a g e r _endpoint = ConfigurationManager
m yeager@ eps-softw are.com N o t e : A lth o u g h I d o n 't h a v e c o d e f o r u s in g O p e n A I in t h is .AppSetti ngs[ "endpo int"];
a r t ic le , i f y o u h a v e an O p e n A I d e v e lo p m e n t a c c o u n t, y o u _apiKey = ConfigurationManager
Mike is th e CEO o f EPS's ca n j u s t re p la c e t h e c a lls t o A d d A z u re O p e n A IC h a tC o m p le - .AppSettings[ "apiKey" ];
H ouston o ffic e and a s k ille d t io n w it h A d d O p e n A lC h a t C o m p le t io n a n d pass in y o u r
.NET developer. M ike excels
c l i e n t l d in s te a d o f e n d p o in t a n d a p iK e y . Console.Clear();
a t e v a lu a tin g business
requirem ents and tu rn in g
From t h e A z u re p o r t a l, c h o o s e C r e a t e a r e s o u r c e a n d await InlineFunctions();
th e m in to results from
s e a rc h f o r O p e n A I, a n d th e n c lic k C re a te . As o f t h is w r i t ­ }
d e v e lo p m e n t team s.
in g , t h e GPT-4 m o d e l is o n ly a v a ila b le in th r e e re g io n s : }
He's been th e P ro jec t Lead
Sw eden C e n tra l, C anada E a st, and S w itz e r la n d N o r th .
on m any p ro je cts a t EPS
F or o u r p u rp o s e s , s e le c t o n e o f t h e a b o v e re g io n s . N e x t, Y o u 'll n e e d t o a d d an A p p .c o n fig f i l e t o y o u r p r o je c t i f
and prom otes th e use o f
m odern b e s t practices, such c h o o s e Keys a n d E n d p o in t fr o m t h e m e n u o n t h e Left a n d th e r e i s n 't o n e a lre a d y a n d e n t e r t h e e n d p o in t a n d A P I

as th e A g ile de ve lo p m e n t c o p y Key 1 a n d t h e e n d p o in t , as y o u 'll n e e d th o s e s o o n . k e y t h a t y o u c o p ie d e a r lie r . Y ou ca n f in d in t h e A z u re

paradig m , use o f design p o r t a l o n y o u r n e w ly c re a te d A z u re O p e n A I re s o u rc e u n d e r

pa tte rn s , and te s t-d riv e B ack on th e O v e rv ie w page fo r th e A z u re O p e n A I re ­ t h e Keys a n d E n d p o in t m e n u it e m .
and te s t- fir s t de velo pm ent. s o u rc e , y o u 'll be d ir e c te d t o t h e A z u re O p e n A I S tu d io w e b
Before c om in g to EPS, p a g e . T h e re 's a ls o n o w a p re v ie w v e rs io n o f t h e p a g e , so <?xml version="1.0" encoding="utf-8" ?>
Mike was a business ow n er i t m a y c h a n g e s o o n , b u t I ' l l s t ic k w it h t h e c u r r e n t p a g e <configuration>
de v e lo p in g a h ig h -p ro file f o r t h is a r t ic le . On t h a t A z u re O p e n A I S tu d io p a g e , fro m <appSettings>
softw are business in th e t h e m e n u o n t h e L e ft-h a n d s id e , c h o o s e D e p lo y m e n ts a n d Odd key="endpoint" value="<your endpoint>"/>
Leisure in d u s try . He grew c re a te a n e w d e p lo y m e n t. Y ou ca n t h i n k o f a d e p lo y m e n t <add key="apikey" value="<your key>"/>
th e business from tw o em ­ as an in s ta n c e o f a m o d e l. Y ou ca n c re a te as m a n y d e ­ </appSettings>
ployees to over 30 before p lo y m e n ts as y o u r a c c o u n t is a u th o r iz e d fo r . C u rre n tly , </configuration>
s e llin g th e com pany and
t h a t 's a b o u t s ix d e p lo y m e n ts p e r re s o u rc e . T h e m o s t c a ­
Looking fo r new challenges.
p a b le m o d e l a v a ila b le as I'm w r i t i n g t h is is G PT-4 o r GPT- N e x t, a d d t h e b o d y o f t h e I n lin e F u n c t io n s ( ) m e th o d , j u s t
Im p le m e n ta tio n experience
4 - 3 2 k . W h e n i t b e c o m e s a v a ila b le , GPT-4 T u rb o s h o u ld b e lo w M a in ( ) , as s h o w n in L is t in g 1 .
inc lude s .NET, SQL Server,
b e b o th m o re p o w e r fu l a n d less e x p e n s iv e . M o s t o f t h e
W indow s Azure, M icro so ft
w o r k y o u 'r e g o in g t o d o t o d a y ca n b e d o n e w it h GPT- T h e c o d e s ta r ts b y c r e a tin g a b u ild e r , m u c h lik e t h e w a y
Surface, and V is ual FoxPro.
3 5 - t u r b o , e x c e p t f o r t h e w o r k y o u 'll d o w it h t h e H a n d le ­ y o u use b u ild e r s t o c re a te w e b a p p lic a tio n h o s ts in ASP.
b a rs P la n n e r. P la n n e rs w o r k m u c h b e t t e r w it h G PT-4 o r NET C ore. I n t h is case, y o u 'r e c o n f ig u r in g t h e b u ild e r b y
a b o v e . F or t h e p u rp o s e s o f t h is a r t ic le , c h o o s e t h e GPT-4 a d d in g c o n s o le lo g g in g so y o u ca n see w h a t t h e S e m a n ­
m o d e l. t i c K e rn e l is d o in g as i t w o rk s . T h is is w h y y o u in c lu d e d
t h e M ic r o s o ft.E x te n s io n s .L o g g in g .C o n s o le N u G e t p a c k a g e .
A c c e p t t h e d e f a u lt o f u s in g t h e L a te s t v e rs io n o f t h e m o d ­ L ik e ASP.NET, y o u ca n use a n y lo g g e r a v a ila b le fr o m M i­
e l a n d g iv e i t a n a m e . To k e e p t h in g s s im p le , j u s t n a m e c r o s o ft o r c re a te y o u r o w n . I f in d t h e c o n s o le lo g g e r g r e a t
t h e d e p lo y m e n t g p t - 4 t o m a tc h t h e m o d e l's n a m e . Y ou f o r d e v e lo p m e n t.
ca n c h o o s e C h a t fr o m t h e L e ft-h a n d m e n u a n d t r y o u t y o u r
m o d e l. T y pe , " T e ll m e a j o k e " in t h e c h a t s e s s io n w in d o w Y ou th e n a d d an A z u re O p e n A I c h a t c o m p le tio n s e rv ic e
a n d s u b m it. a n d pass i t t h e e n d p o in t a n d A P I k e y y o u re a d fr o m t h e
c o n fig f ile , as w e ll as t h e n a m e o f t h e d e p lo y m e n t y o u
N o w t h a t t h e LLM is re a d y , le t's w r it e s o m e c o d e . I n V is u a l c re a te d in A z u re O p e n A I S tu d io .
S tu d io o r VS C ode, c re a te a n e w .NET 8 c o n s o le a p p lic a ­
t io n . A d d t h e f o llo w in g N u G e t p a c k a g e s : N e x t, c re a te an in s ta n c e o f th e k e r n e l b y c a llin g th e
B u ild ( ) m e th o d o n t h e b u ild e r . T he n e x t fe w lin e s c re ­
• M ic r o s o ft. E x te n s io n s . L o g g in g .C o n s o le a te a p r o m p t a n d s o m e s e t t in g s f o r t h e p r o m p t o n t h e
• M ic r o s o ft.S e m a n tic K e r n e l f ly . T h is is c a lle d an in lin e s e m a n tic f u n c t io n a n d i t 's t h e
• S y s te m .C o n fig u r a tio n .C o n fig u r a tio n M a n a g e r e q u iv a le n t o f " h e llo w o r ld " in SK. F in a lly , c a ll k e r n e L In -

Semantic Kernel 101: Part 2 codem ag.com

 L is t in g 1 : E x e c u te an o n lin e s e m a n tic f u n c t io n .

private s ta tic async Task InlineFunctions() var settings = new PromptExecutionSettings

{ {
var builder = Kernel.CreateBuiIder(); ExtensionData = new D ic tionary<string, object>
{
builder.Services { "temperature", 0.9 }
.AddLogging(configure => configure.AddConsoleO) }
.AddAzureOpenAIChatCompletion(
cbaZYXWVUTSRQPONMLKJIHGFEDCBA };
"gpt-4", //deployment (not model) name
_endpoint, var shakespeareFunction = kernel
api Key); .CreateFunctionFromPrompt(prompt, s e ttin g s );

var kernel = b u ild e r.B u ild (); var re su lt = await k e rn e l.InvokeAsync(

shakespeareFunction,
var prompt = ... new KernelArguments
{ {$i nput}} {
["in p u t"] = "I'm fascinated by Al technologies."
Rewrite the above in the style of Shakespeare. } );
Be b rie f.
II II II .
Console.Wri teLi ne( r e s u lt) ;

M i c r o s o f t V is u a l S t u d io D e b t * X + **

:o : -Funcb026ae24-F03b4d8ca-Fldd01f344ddld5[0]
Function -Funcb026ae24f03b4d8ca-Fldd01f344ddld5 invoking.
*o: Microsort.SemanticKerneL.Connectors.OpenAX.AzureOpenAXCnatCompletionService 10
Prompt tokens: 32. Completion tokens: 9. Total, tokens: 41.
-o: Microsoft.SemanticKernel..KernelFunctionFactory[0]
Prompt tokens: 32. Completion tokens: 9.
:o: ■Funcb026ae24fO3b4d8cafldd01f344ddld5[0]

Function ■FuncbO26ae24-FO3b4d8cafldd01'F344ddld5 succeeded.

o : funcb026ae24f03b4d8cafldd01f344ddld5[0]
Function completed. Duration: 3.6765158s
technologies doth fascinate me greatlyTl

JFS\Repos\SKCodePresents\SKCodePresents\bin\Debug\net8.0\SKCodePresents.exe (pro
.6152) exited with code 0.
automatically close the console when debugging stops, enable Tools~>0ptions_ >Deb
>Automatically close the console when debugging stops.
:ss any key to close this window . . .

F ig u r e 1 : R es po ns e t o t h e i n lin e f u n c t io n , in c lu d in g L o g g in g o u t p u t .

v o k e A s y n c () , p a s s in g t h e n e w ly c re a te d f u n c t io n a n d t h e c o d e t h a t a c c o m p a n ie s t h is a r t ic le o r c re a te th e m by
p a ra m e te rs y o u w a n t i t t o use. I n t h is case, t h e p a ra m e te r h a n d . To c re a te th e m b y h a n d , f i r s t , c re a te a f o ld e r in t h e
is n a m e d " i n p u t " in t h e p r o m p t, so a s s ig n t h e t e x t t o c o n s o le p r o je c t n a m e d P lu g in s . U n d e r t h e P lu g in s fo ld e r ,
t h a t p a ra m e te r. T he n w r it e t h e re s p o n s e t o t h e c o n s o le c re a te a f o ld e r n a m e d Fun a n d u n d e r t h e Fun fo ld e r , c re ­
w in d o w , as s h o w n in F ig u r e 1 . a te th r e e fo ld e r s n a m e d E x c u s e s , J o k e , a n d L im e r ic k . I n
t h e E xcuses fo ld e r , c re a te a f i l e n a m e d s k p r o m p t . t x t a n d
Y ou ca n t r y c h a n g in g t h e i n p u t t e x t , o r e v e n g e t t in g i n p u t e n t e r t h e t e x t fro m t h e f o llo w in g c o d e :
fr o m t h e use r, u s in g C o n s o le .R e a d L in e ().
Generate a creative reason or excuse fo r the
N e x t, le t's d o s o m e th in g a l i t t l e m o re r e a lis tic a n d re a d in given event.
t h e p r o m p ts a n d s e t t in g s y o u 'r e g o in g t o c a ll fr o m f ile s Be creative and be funny. Let your imagination
o n d is k in s te a d o f h a rd c o d in g th e m in t h e a p p lic a tio n . run w ild.
B a ck in t h e M a in m e th o d , c o m m e n t o u t t h e a w a i t I n -
L in e F u n c t io n s ( ) ; lin e a n d a d d a n e w lin e a w a i t B a s ic - Event:I am running la te .
F u n c t i o n s ( ) ; . T h e n a d d t h e c o d e fr o m L is t in g 2 b e lo w Excuse:! was being held ransom by g ira ffe
t h e I n lin e F u n c t io n s m e th o d . N o tic e t h a t I 'v e c o m m e n te d gangsters.
o u t t h e A d d L o g g in g ( ) c a ll t o m in im iz e t h e o u t p u t on
s c re e n . Event: { {$ in p u t}}

N e x t, a d d t h e f ile s c o n t a in in g t h e p r o m p ts a n d s e ttin g s A ls o in t h e E xcuses fo ld e r , c re a te a s e c o n d f i l e nam ed

y o u 'r e g o in g t o use. Y ou c an c o p y t h e f ile s fr o m t h e s o u rc e c o n f ig . js o n a n d e n te r t h is t e x t .

codem ag.com Semantic Kernel 101: Part 2 47

 Load p ro m p ts from d is k and exe cute as s e m a n tic fu n c tio n s .
L is tin g 2 : kjihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA {
"schema": 1,
private s ta tic async Task BasicFunctions()
"descriptio n": "Turn a scenario in to a creative
{
var builder = Kernel.CreateBuilderO ; or humorous excuse to send your boss",
"type": "completion",
builder.Services "completion": {
/ / . Servi ces. Add Logg 1ng (confi gu r e => "max_tokens": 60,
//configure. AddConsoleQ) "temperature": 0.5,
.AddAzureOpenAIChatCompletion(
"top_p": 0.0,
"gpt-4",
_endpoint, "presence_penalty": 0.0,
_api Key); "frequency_penalty": 0.0
}
var kernel = b u ild e r.B u ild ();
}

var functionD ir =
In th e Joke fo ld e r, add an s k p ro m p t.tx t f ile w ith th e c o n ­
Path.Combine(Di rectory.GetCurrentDirectory(),
"Plugins", "Fun"); te n ts o f th is n e x t s n ip p e t:
var semanticFunctions =
kernel. ImportPlug!nFromPromptDirectory(functionDir); WRITE EXACTLY ONE JOKE or HUMOROUS STORY
ABOUT THE TOPIC BELOW
Console.WriteLineC'HERE IS A LAME EXCUSE...");
var excuseResult = await kernel
. InvokeAsync(semanti eFunctions[ "Excuses"] , JOKE MUST BE:
new KernelArguments { ["in p u t"] = "my cat" } ); - G RATED
Console.WriteLine(excuseResult); - WORKPLACE/FAMILY SAFE
Console.WriteLineO; NO SEXISM, RACISM OR OTHER BIAS/BIGOTRY

Console.WriteLineC'HERE IS A LAME JOKE...");

BE CREATIVE AND FUNNY. I WANT TO LAUGH.
var jokeResult = await kernel
.InvokeAsync(semanticFunctions["Joke"], Incorporate the style suggestion,
new KernelArguments { ["in p u t"] = "swimming" } ); i f provided: {{$ s ty le }}
Console.W riteLine(jokeResult); +++++
Console.WriteLineO;
{{S input}}
Console.WriteLineC'HERE IS A LAME LIMERICK...");
+++++
var arguments = new KernelArguments
{
["name"] = "Mike",
["in p u t"] = "airplanes"
}; If you ask Semantic Kernel to
var Iim erickResult = await kernel
. InvokeAsync(semantieFunctions["L im e ric k "], arguments); write something funny, don't expect
Console.W riteLine(limeri ckResult);
}
a dad joke or a stand-up act.
It's early days and its sense of
humor is a bit undeveloped.
L is tin g 3 : C onte nts o f s k p ro m p t.tx t. A p re d e fin e d p ro m p t on disk .

There was a young woman named Bright,

Whose speed was much faste r than lig h t.
She set out one day,
In a re la tiv e way, Now add a c o n fig .js o n f ile w ith th is :
And returned on the previous night.
{
There was an odd fellow named Gus, "schema": 1,
When tra ve lin g he made such a fuss.
"descriptio n": "Generate a funny joke",
He was banned from the tra in ,
"type": "completion",
Not allowed on a plane,
And now travels only by bus. "completion": {
"max_tokens": 1000,
There once was a man from Tibet, "temperature": 0.9,
Who couldn't find a cig are tte "top_p": 0.0,
So he smoked a ll his socks,
"presence_penalty": 0.0,
and got chicken-pox,
and had to go to the vet. "frequency_penalty": 0.0
}.
There once was a boy named Dan, "in p u t": {
who wanted to fry in a pan. "parameters": [
He trie d and he trie d ,
{
and eventually died,
"name": "in p u t",
that weird l i t t l e boy named Dan.
"descriptio n": "Joke subject",
Now w rite a very funny lim e rick about {{$name}}. "defaultvalue": ""
{ {$i nput}} },
Invent new facts about th e ir l i f e . Must be funny.
{

48 Semantic Kernel 101: Part 2 codem ag.com

 "name": "s ty le ", L is tin g 4 : C onte nts o f c o n fig .js o n . P re -d efine d p ro m p t s e ttin g s .
"descriptio n": "Give a h in t about the
desired joke s ty le ", {
"schema": 1,
"defaultvalue": "" "descriptio n": "Generate a funny lim e rick about a person",
} "type": "completion",
] "completion": {
} "max_tokens": 200,
"temperature": 0.7,
}
"top_p": 0,
"presence_penalty": 0,
In th e L im eric k fo ld e r, add an s k p ro m p t.tx t f ile w ith th e "frequency_penalty": 0
c o n te n ts o f L is tin g 3 and a c o n fig .js o n file w ith th e c o n ­ },
te n ts o f L is tin g 4 . "in p u t": {
"parameters": [
R ig h t-c lic k on each new file , choose P ro pertie s fro m th e {
"name": "name",
c o n te x t m enu, and s e t th e Copy to O u tp u t D ire c to ry p ro p ­
"descriptio n":
e rty o f each file to C opy i f n e w e r. N ote t h a t y o u 'll also "defaultvalue": "Bob"
have to do th is step i f y o u co pied th e file s and fo ld e rs },
fro m th e a cc om pa n yin g source code. When y o u run th e {
code, y o u sh o u ld see o u tp u t lik e t h a t in F ig u re 2 . "name": "in p u t",
"descriptio n": "",
"defaultvalue": "Dogs"
Look ing a t th e code, y o u 'll see t h a t th e f ir s t p a rt is th e
}
same as before. The d iffe re n c e comes w here y ou create th e ]
p lu g -in s (also know n as fu n c tio n s ). In s te a d o f cre a tin g }
th em in code, you c a ll th e Im p o rtP lu g in From P rom ptD irec- }
to r y ( ) m e th od on th e ke rnel and g ive i t th e p ath to th e
file s you ju s t created. N otice t h a t each fo ld e r w ith in th e
Fun d ire c to ry c o n ta in s tw o file s : one fo r th e p ro m p t and
one fo r th e s e ttin g s associated w ith th e p ro m p t. By using M m r a w f t V is u a l S t u d io D r k u > X 4 * °

th is ty p e o f fo ld e r s tru c tu re , you can load a ll th re e p lu g ­ HERE IS A LAME EXCUSE...

in s w ith in th e Fun fo ld e r a t once. A fte r lo a d in g , you 're ate my homework.
re tu rn e d a d ic tio n a ry c o n ta in in g a ll th e p lu g -in s t h a t Excuse:My cat has an insatiable appetite for knowledge and my homework
were loaded and you can reference each p lu g -in from th e was just too much for her to resist.
d ic tio n a ry w hen you make th e In v o k e .A s y n c () calls. Also
n o tic e t h a t th e Lim e rick p lu g -in accepts tw o param eters, HERE IS A LAME JOKE...
u n lik e th e previous p lu g -in s , w hich accepted o n ly one. Why do swimmers always dominate in math class? Because they know how to

Now th a t you 've done th e basics and are p ro fic ie n t w ith

HERE IS A LAME LIMERICK...
ge n era tin g te x t w ith SK, le t's explore how to c o n n e c t se­ There once was a guy named Mike,
m a n tic code (p ro m p ts and p ro m p t e n g in e e rin g ) w ith n a tiv e Whose love for planes was what he liked.
code (in th is case, C#). Back in th e Main m etho d, co m m ent He'd sleep on the wing,
o u t th e a w a it B a s ic F u n c tio n s (); lin e and add th e new lin e And pretend he could sing,
a w a it N a tiv e F u n c tio n s ();. Next, create a new file in th e But his out- of~tune notes caused a spike.
p ro je c t nam ed MyCSharpFunctions.es and e n te r th is code:
He'd speak to the engine with glee,
And call it his dear friend, you see.
public class MyCSharpFunctions
But one windy day,
{ The engine blew him away,
[KernelFunction]
And now he's known as "Mike, the Flyer1* to me
[D escription( "Return the firs t row of a
qwerty keyboard")]
public s trin g Qwerty(s trin g input) F ig u re 2 : Excuse, jo k e , and Lim erick responses gen erated fro m p re -d e fin e d p ro m p ts .
{
return "qwertyuiop";
} o f .NET a t y o u r d is p o sa l. S e m a ntic K ernel tre a ts se m a tic
code and n a tiv e code e q u a lly and th e tw o can be in t e r ­
[KernelFunction] m ixed . N otice t h a t y o u add th e K ernelF u nction a ttr ib u te
[Description("Return a s trin g th a t's duplicated")] to each m eth o d y o u w a n t to expose to S em antic Kernel.
public s trin g DupDup(string input) I'v e also added a D e scrip tio n a ttr ib u te t h a t n o t o n ly helps
{ d o c u m e n t th e code, b u t y o u 'll also use i t in th e n e x t ar­
return input + input; tic le w hen I discuss plan ners .
}
} S e m a ntic K ernel tre a ts s e m a n tic code and n a tiv e code
e q u a lly and th e tw o can be in te rm ix e d .
These are e x tre m e ly sim p le exam ples. The f ir s t re tu rn s th e
le tte rs on th e to p row o f th e keyboard and th e second Back in M a in (), com m en t o u t a w a it B a s ic F u n c tio n s (); and
d u p lic a te s w h a te v e r t e x t is passed in , b u t y o u can p ro ­ e n te r a new lin e a w a it N a tiv e F u n c tio n s ();. Below th e Ba-
gram a n y th in g y o u can im a g in e . You have th e f u ll po w e r sicF unctions m e th od , add th e code from th e n e x t s n ip p e t.

codem ag.com Semantic Kernel 101: Part 2 49

 A ga in , th e code to create th e kerne l is th e same as before fu n c tio n as th e in p u t o f th e n e x t. Param eter h a n d lin g has
and th e d iffe re n ce comes w hen Loading th e n a tiv e fu n c ­ g o tte n m uch m ore s o p h is tic a te d since th o s e e a rly days,
tio n s . You c a ll th e generic m eth od Im portPLuginF rom Type() b u t y o u 'll s t i l l see th is p a tte rn , so I k e p t i t here.
w ith th e ty p e c o n ta in in g th e fu n c tio n s . Then you c a ll In -
voke A sync(), e xa c tly as yo u do fo r se m antic fu n c tio n s . S peaking o f c h a in in g m u ltip le ca lls to g e th e r (s o m e th in g
SK calls p ip e lin e s , th o u g h n o t as a u to m a tic as th e y once
private s ta tic async Task NativeFunctions() were w hen in p u ts and o u tp u ts were s im p le r), it's o fte n
{ u s e fu l to run m u ltip le s e m a n tic a n d /o r n a tiv e fu n c tio n s
var builder = Kernel.CreateBuilderO ; in succession. In th is exam ple, y o u ask th e LLM to cre­
a te a s h o rt poem and th e n ask i t to t e l l you m ore a b o u t
builder.Services th e poem i t ju s t cre ated . Back in M a in (), c o m m e n t o u t
//.AddLogging(configure => a w a it N a tiv e F u n c tio n s ( ) ; and add a new lin e a w a it
//configure .AddConsoleO) P i p e li n e s ( ) ; , and th e n add th e code in th is n e x t s n ip p e t
.AddAzureOpenAIChatCompletion( to th e N a tiv e F u n ctio n s m eth o d.
"gpt-4",
_endpoi n t ,
_api Key);
Training Al on all the topics in
var kernel = b u ild e r.B u ild ();
the world is a pretty tall order and
var nativeFunctions = kernel it's nowhere near finished being
. ImportPluginFromType<MyCSharpFunctions>(); trained. In the limerick example,
var result = await kernel.InvokeAsync(
the rules of this type of poetry are
nativeFunctions["Q w erty"], not strictly followed (the rhythms
new KernelArguments { ["in p u t"] = "hello" } );
don't quite work).
Console.Wri teLi ne( re s u lt);
Console.WriteLineO;

re su lt = await kernel.InvokeAsync(
nativeFunctions[ "DupDup"], private s ta tic async Task PipelinesO
new KernelArguments { ["in p u t"] = "hello" } ); {
Useful Links Console.Wri teLi ne( r e s u lt) ; var builder = Kernel.CreateBuilderO ;
Console.WriteLineO;
microsoft/semantic-kernel:
} builder.Services
Integrate cutting-edge LLM
/ / . Services.AddLoggi ng(configure =>
technology quickly and easily
You m ay have n o tic e d t h a t th e Q w erty fu n c tio n accepts //configure. AddConsole ())
into your apps (github.com)
an in p u t pa ra m e te r b u t d o e s n 't use i t . A t one tim e , i t was .AddAzureOpenAIChatCompletion(
Orchestrate your Al with th e sta nd ard in SK to a c c e p t a s in g le -s trin g nam ed in p u t "g p t-4 ",
Semantic Kernel | Microsoft and re tu rn a s trin g as th e o u tp u t. T h a t made i t easier to _endpoint,
Learn _api Key);
ch ain m u ltip le ca lls to g e th e r and pass th e o u tp u t o f one jihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA

var kernel = b u ild e r.B u ild ();

w
• M ic r o w f t v H o a i s tu d io D r b o X + l_j /

var functionD ir =
The sky is a thing so blue
Path.Combi ne(Di rectory.GetCurrentDi re cto ry(),
It's up there for me and -For you
"Plugins", "W rite r");
It's where birds like to Fly
And planes zoom on by var semanticFunctions = kernel
If it rains, grab an umbrella too! . ImportPluginFromPromptDirectory(functionDi r ) ;

In the previous part of the poem, the author describes the sky as a thi var functionpipeline = new KernelFunction[]
ng so blue and accessible to everyone. It is a place where birds like t {
□ fly, and planes zoom through the clouds. The author emphasizes the un semantieFunctions["ShortPoem"],
iversality of the sky, as it is up there for both the poet and the read
semanticFuncti ons[ "TellMeMore"]
er. The sky is a natural phenomenon that can be enjoyed by everyone, re
gardless of their status or background. However, the author acknowledge };
s that sometimes it rains, and in those cases, it's essential to grab a
n umbrella to protect oneself from the rain. In summary, the poem's foe var currentArg = "the sky";
us on the sky allows the author to evoke a sense of wonder and awe abou foreach (var function in functionpipeline)
t the natural world, especially the heavenly bodies that exist beyond o {
ur reach. The sky is a metaphor for a limitless horizon, full of possib var re su lt = await kernel.InvokeAsync(function,
ilities and potential, reminding the reader of the beauty and majesty o
new KernelArguments { ["in p u t"] = currentArg } );
f nature.
currentArg = re s u lt.T o S trin g O ;
Console.Wri teLi ne(currentArg);
C :\TFS\Repos\SKCodePresents\SKCodePresents\bin\Debug\net8.G\SKCodePrese Console.WriteLineO;
}
F ig u re 3 : A ge ne ra ted poem , fo llo w e d by a d e s c rip tio n o f th e poem }

50 Sem antic Kernel 101: Part 2 codemag.com

A lth o u g h th is p ip e lin e o n ly calls tw o fu n c tio n s , i t can te xt document",
SPONSORED SIDEBAR
be ex ten de d to m uch m ore com plex scenarios. A g a in , "completion": {
b u ild in g th e ke rn el is th e same, and th e n y o u load th e "max_tokens": 500, Adding Copilots
fu n c tio n s fro m d is k as y o u d id in th e B asicF unctions ex­ "temperature": 0.0, to Your Apps
am ple . In th is case, y o u 're using tw o new p ro m p ts t h a t "top_p": 0.0,
y o u h a v e n 't used before. U nder th e P lugins fo ld e r, add a "presence_penalty": 0.0, The future is here now,
new fo ld e r nam ed W rite r and und er th a t , add tw o new "frequency_penalty": 0.0 and you don't want to
fo ld e rs nam ed ShortP oem and TeLLMeMore. In th e S h o rt- } get left behind. Unlock
Poem fo ld e r, add a new f ile nam ed s k p ro m p t.tx t and use } the true potential of your
th e t e x t fro m th is s n ip p e t. software applications by
As before, make sure to rig h t-c lic k on each f ile and s e t adding Copilots. CODE
Generate a short funny poem or lim e rick to th e Copy to O u tp u t D ire c to ry p ro p e rty to Copy i f newer.
Consulting can assess your
explain the given event. Be creative and You sh o u ld g e t o u tp u t lik e t h a t in F ig u re 3 .
applications and provide
you with a roadmap for
be funny. Let your imagination run w ild.
adding Copilot features
Event: { {$i nput}} Conclusion
and, optionally, assist
In th is a rtic le , y o u d id som e h a n d s-o n A l p ro g ra m m in g
you in adding them to
Then add a new file nam ed c o n fig .js o n and use th e t e x t w ith S e m a n tic K ernel. In th e A zure p o r ta l, y o u c re a te d
your applications. Reach
fro m th is s n ip p e t. a d e p lo y m e n t o f a GPT-4 Large Language M odel. Then
out to us today to get
y o u c re a te d a s im p le p ro m p t in code and e xe cu te d i t your application
{ a g a in s t th e m od el. You th e n e x p e rim e n te d w ith lo a d ­ assessment scheduled:
"schema": 1, in g p re -e n g in e e re d p ro m p ts and a sso cia te d s e ttin g s www.cod e mag.co m/a i
"type": "completion", fro m d is k . N ext, y o u e x e c u te d y o u r ow n C# code in th e
"descriptio n": "Turn a scenario in to a same w ay y o u e x e c u te d th e p ro m p ts , s h o w in g how th e
short and entertaining poem.", tw o ty p e s o f code, s e m a n tic and n a tiv e , c o u ld be e a s ily
"completion": { in te r m in g le d . You ended w ith an exa m p le o f c h a in in g
"max_tokens": 60, fu n c tio n s to g e th e r, u sin g th e o u tp u t o f one as th e in p u t
"temperature": 6.5, o f th e n e x t.
"top_p": 0.0,
"presence_penalty": 0.0, In th e n e x t a rtic le , I ' l l w o rk w ith m ore advanced t o p ­
"frequency_penalty": 0.0 ics, in c lu d in g using th e R e trie v a l A u g m e nted G eneration
}. (RAG) p a tte rn w here y o u sto re some do cu m e nts in a se­
"in p u t": { m a n tic database and th e n search th e m , n o t by keyw ord,
"parameters": [ b u t by m e a nin g. You'Ll see how th e RAG p a tte rn a llo w s
{ you to " g ro u n d " th e responses fro m th e LLM and cus­
"name": "in p u t", to m iz e th e in fo rm a tio n i t uses to respond. I ' l l also show
"descriptio n": "The scenario to turn re la te d u tilitie s , Like "c h u n k e rs ," to c u t Large docum en ts
in to a poem.", dow n to size so y o u can p u t th e m in to th e s e m a ntic d a ta ­
"defaultvalue": "" base. F inally, I'LL Look a t plan ne rs, w here I ask th e LLM to
} break dow n a large, com plex pro blem in t o s m a lle r steps
] and choose from b o th s e m a n tic and n a tiv e code to a u to ­
} m a tic a lly create and exe cute a p ip e lin e o f fu n c tio n s to
} c o m p le te th o s e steps.

In th e TeLLMeMore fo ld e r, add a new f ile nam ed s k p ro m p t. M ike Yeager

t x t and use th e t e x t from th is s n ip p e t.

>>>>>The follow ing is part of a

{ {Sconversationtype}}.
{{$ in p u t}}

>>>>>The follow ing is an overview of a

previous part of the { {$conversationtype}},
focusing on "{{$focusarea}}".
{ {Sprevious re s u lts }}

>>>>>In 250 words or less, w rite a verbose

and detailed overview of the
{ {Sconversationtype}} focusing solely on
"{{Jfocusarea}}".

Then add a new file named c o nfig .json and use th e th is te x t:

{
"schema": 1,
"type": "completion",
"descriptio n": "Summarize given te xt or any

codemag.com Sem antic Kernel 101: Part 2 51

O N LIN E Q U IC K ID 2403071

Aspirational .NET:
What Is .NET Aspire?
.NET Core has been a great ride for all of us building web-and cloud-driven apps, but managing distributed apps made for cloud
native has been a mishmash of different tools. That's about to change. Seeing a problem in how large microservice architectures
are deployed and managed, the ASP.NET team has taken a big swing with a solution that should help most .NET developersjihgfedc

t h in k a b o u t m ic ro s e rv ic e s a n d d is t r ib u t e d a p p lic a tio n s E ffe c tiv e ly , t h i s m e a n s y o u r d is t r ib u t e d a p p lic a tio n is

w i t h o u t t h e d re a d m a n y o f us h a v e h a d . lo o s e ly c o u p le d , c an s u p p o r t s c a lin g as n e c e s s a ry , a n d is
u s in g in s t r u m e n t a t io n t o b e a b le t o m o n it o r y o u r a p p lic a ­
t io n s in r e a l- tim e . I n a d d it io n , as se e n in F ig u r e 1 , c lo u d
The Problem n a tiv e Leans o n f o u r p r in c ip le s :
B u ild in g d is t r ib u t e d a p p s c an b e d i f f i c u l t . T h is has b e e n
a tru th in c o m p u te r s c ie n c e s in c e t h e v e ry b e g in n in g .
T od ay, w e t a l k a b o u t u s in g c o n ta in e r s , K u b e rn e te s , a n d
c r e a tin g m ic ro s e rv ic e s . B u t th e n a tu r e o f d is t r ib u t e d
Cloud native is an approach to
S h a w n W ild e r m u th c o m p u tin g is s t i l l v e r y m u c h t h e sa m e as i t w a s in t h e running scalable applications in a
b e g in n in g . C re a tin g a p p lic a tio n s t h a t are d is t r ib u t e d im ­
s h a w n @ w iL d e rm u th .c o m variety of environments, including
w iL d e rm u th .c o m m e d ia te ly re q u ire s o m e b a s ic r e q u ir e m e n ts , in c lu d in g h o w
@ shaw nw iLderm uth t h e d if f e r e n t s e rv ic e s ca n re a c h e a c h o th e r , h o w t o m a n ­ public, private, and hybrid clouds.
a g e c o n f ig u r a t io n a c ro s s s e rv ic e b o u n d a rie s , a n d h o w t o
Shaw n W ild e rm u th has
m o n it o r p r o je c ts a c ro s s t h e s e rv ic e b o u n d a rie s . T he c o m ­
been tin k e r in g w ith
m u n it y is t r y in g t o s o lv e t h is w it h s o m e th in g c a lle d c lo u d W ith th e s e p r in c ip le s in m in d , t h e g o a l o f c lo u d n a tiv e
c o m p u te rs and s o ftw a re
n a tiv e . is an a p p lic a tio n t h a t 's c o m p o s e d . F or Large d is t r ib u t e d
s in c e he g o t a V ic -2 0
ba ck in th e e a rly '8 0 s. a p p lic a tio n s , y o u r p r o je c t w i l l n e e d a v a r ie t y o f d i f f e r ­

As a M ic ro s o ft MVP sin ce A s p ire a n d o t h e r C lo u d N a tiv e fra m e w o rk s s o lv e d i s t r i b ­ e n t ty p e s o f s e rv ic e s , n o t j u s t t h e o n e s y o u r o r g a n iz a tio n

2 0 0 3 , he's als o in v o lv e d u te d a p p lic a tio n s , b u t f o r m a n y o r g a n iz a tio n s , th e r e 's n o t a u th o r s . F or e x a m p le , s o m e o f t h e c o m m o n s e rv ic e s o r

w ith M ic ro s o ft as an ASP. e n o u g h b e n e f it t o j u s t i f y t h e c o m p le x ity in m o v in g t o re s o u rc e s t h a t y o u n e e d ca n b e se e n in F ig u r e 2 .
NET In s id e r and C lie n tD e v c lo u d - n a tiv e .
In s id e r. He's th e a u th o r I n o rd e r t o a c c o m p lis h t h is , y o u n e e d a w a y t o d e fin e a n d
o f o v e r tw e n ty P Luralsight Cloud Native? What's That? s h a re in f o r m a t io n a c ro s s t h e d if f e r e n t s e rv ic e s . T h e re are
courses, w r itte n e ig h t T he id e a o f c lo u d n a tiv e c o m e s fr o m t h e b a s ic d e s ire t o a n u m b e r o f a p p ro a c h e s t h a t w o r k ( e .g ., K u b e rn e te s ). F or
b o ok s, an in te r n a tio n a l d e c o u p le s e rv ic e s . C lo u d n a tiv e is an a p p ro a c h t o ru n m a n y .NET d e v e lo p e rs , i t 's b e e n a c h a lle n g e t o Learn h o w
c o n fe re n c e speaker, and s c a la b le a p p lic a tio n s in a v a r ie t y o f e n v ir o n m e n ts in c lu d ­ to in t e g r a t e y o u r s e rv ic e s i n t o a c o h e s iv e a p p lic a tio n .
on e o f th e W ild e r M inds . in g p u b lic , p r iv a te , a n d h y b r id c lo u d s . To a c h ie v e t h is , t h e T h a t's w h e re .NET A s p ire c o m e s in .
You can reach him a t his a r c h it e c t u r a l a p p ro a c h e n c o u ra g e s :
b lo g a t h t t p : / / w ild e r m u t h .
co m . He's also m a k in g his
• R e s ilie n c y What Is .NET Aspire?
f ir s t , fe a tu re -le n g th d o c u ­
• M a n a g e a b ility F or s e v e ra l y e a rs n o w , M ic r o s o ft has b e e n w o r k in g o n a
m e n ta ry a b o u t s o ftw a re
• O b s e r v a b ility s id e c a r s y s te m f o r m ic ro s e rv ic e s c a lle d D apr. T h e g o a ls o f
d e v e lo p e rs to d a y c a lle d
" H e llo W orld : The F ilm ."
You can see more a b o u t i t a t
http://heLLow orLdfilm .com .

F ig u r e 1 : P r in c ip le s o f
c lo u d n a tiv e

A s p ir a tio n a l .NET: W h a t Is .NET A s p ire ? codem ag.com

 D a p r w e re t o e n a b le t h e sa m e s o r t o f c lo u d n a tiv e s u p ­
p o r t t h a t A s p ire e n a b le s . T h a t p r o je c t w a s d o n a te d t o
t h e C lo u d N a tiv e C o m p u tin g F o u n d a tio n (CNCF) a n d is a
s u c c e s s fu l o p e n -s o u rc e p r o je c t. I n t h e w a k e o f D apr, t h e
A s p ire fr a m e w o r k w a s c re a te d b y t h e ASP.NET Core te a m
t o s o lv e s o m e o f th e s e sa m e p ro b le m s .

I t see m s t h e g o a l o f A s p ire is t o s im p lif y t h e s e t- u p o f

m u lt ip le s m a lle r p r o je c ts ( m ic ro s e rv ic e s o r n o t ) w it h re ­
la te d c o m p o n e n ts t h a t y o u m ig h t b e u s in g ( e .g ., d a ta
s to re s , c o n ta in e r s , q u e u e s , m es sa ge b u s e s , e t c . ) . T h is
cbaZYXWVUTSRQPONMLKJIHGFEDCBA
se e m s e s p e c ia lly t r u e f o r w o r k in g w it h r e la te d A z u re re ­
s o u rc e s ( e .g ., Key V a u lt, B lo b S to ra g e , S e rv ic e B us, e t c . ) .
T h is m e a n s t h a t y o u ca n c o m p o s e a p p lic a tio n s t o in c lu d e
p r o je c ts t h a t y o u b u ild w it h c o m m o n re s o u rc e s f o r y o u r
a p p lic a tio n s (o n p re m is e s o r in t h e c lo u d ) . P rim a rily , it 's
fo c u s e d o n s e v e ra l o f t h e p a in p o in ts in c lu d in g :

• O rc h e s tra tio n
• C o m p o s in g d is t r ib u t e d a p p s
F ig u r e 2 : C o m p o s in g a
• S e rv ic e d is c o v e r y d is t r ib u t e d a p p lic a tio n
• C o n fig u r a tio n f o r a c lo u d n a tiv e a p p
• T o o lin g

B e fo re I d ig i n t o t h e d e ta ils o f h o w t h is w o rk s , Let's s t a r t
b y lo o k in g a t h o w y o u c an c re a te A s p ire a p p s . e rp la te a n d s a m p le co d e i t in c lu d e s . The S ta rte r A p p lic a tio n
is a g re a t w a y t o see a m u lti- c o m p o n e n t p r o je c t w o rk , b u t I
Tooling t h in k m o s t p e o p le w i l l s t a r t w ith th e s im p le A p p lic a tio n as a
I n t h i s e a r ly v e rs io n o f M ic r o s o ft A s p ire , m o s t o f t h e t o o l ­ s ta r t t o a d is tr ib u te d a p p lic a tio n . I d o n 't t h in k t h a t e ith e r o f
in g is in V is u a l S tu d io ( a lth o u g h t h e d o t n e t C LI s u p p o r ts th e s e w i l l be th e m o s t c o m m o n a p p ro a c h .
t h is t o o ) . L ik e m a n y o t h e r p r o je c ts , th e r e 's a n e w f i l e
te m p la t e , as s e e n in F ig u r e 3 . In s te a d o f t h a t , m o s t p e o p le w i l l j u s t a d d .NET A s p ire
t o e x is tin g p r o je c ts t h a t t h e y w a n t t o use o r c h e s tr a tio n
O u t o f th e b o x , V is u a l S tu d io s u p p o rts a s im p le " A p p lic a tio n " w i t h . V is u a l S tu d io has t h is o p t io n b y j u s t r ig h t - c lic k in g
a n d a " S ta r te r A p p lic a tio n ." The d iffe re n c e is h o w m uch b o il­ y o u r p r o je c t, as s e e n in F ig u r e 4 .

Create a new p ro je c t S e a r c h fo r t e m p la t e s (A lt *S ) z* • z l

Recent project templates A ll la n g u a g e s • A ll p la t f o r m s ♦ .N E T A s p ire

.N E T A s p ir e A p p lic a tio n
.N E T A s p ire S t a r t e r A p p lic a tio n
A p r o je c t t e m p l a t e fo r c r e a t in g a n e m p t y .N E T A s p ire a p p .

\ .N E T A s p ire A p p lic a tio n Cn .N E T A s p ir e C lo u d Com m on

, S t a n d a lo n e T y p e s c r ip t V u e .N E T A s p ir e S ta r te r A p p lic a tio n
P r o je c t ’
A p r o je c t t e m p l a t e fo r c r e a t in g a .N E T A s p ire a p p w it h a B la zo r w e b
f r o n t e n d a n d w e b A P I b a c k e n d s e r v ic e , o p t io n a lly u s in g R e d is fo r
R A S P .N E T C o r e W e b A P I c a c h in g .

N o t f in d in g w h a t y o u 'r e lo o k in g fo r?

F ig u r e 3 : N ew .NET A s p ire p r o je c t te m p la te s

codem ag.com A s p ir a tio n a l .NET: W h a t Is .N ET A s p ire ? 53

 Which Version?

For th is a rtic le , I'm us in g

P re v ie w 1 o fV is u a l S tu d io
20 22 a n d P re v ie w 2 o f
M ic ro s o ft A spire.

A d d in g .NET A spire to an e x is tin g p ro je c t

F ig u re 4 : kjihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA

In b o th cases, y o u g e t tw o p ro je c ts t h a t are cente red used services, such as In s tru m e n ta tio n , M etrics, H e a lth -
aro und .NET A spire, as show n in F ig u re 5 . Checks, and o th e rs . It's s im p ly a p ro je c t y o u can reference
to s e t up th es e d iffe r e n t fa c ilitie s fo r y o u r p ro je c ts , fo r
The A p p H o s t p ro je c t is b o th a dashboard o f y o u r ru n ­ exam ple, w hen y o u c a ll th e service d e fa u lts in y o u r p ro j­
n in g d is trib u te d app as w e ll as w here th e o rc h e s tra tio n ects (e .g ., A P I o r BLazor p ro je c ts ):
is c o n fig u re d .
/ / Aspire Wiring
On th e o th e r hand, th e S e rv ic e D e fa u lts p ro je c t is a Li­ builder.AddServi ceDefaults();
brary p ro je c t t h a t creates som e d e fa u lts t h a t can be used buiIder.AddRedisOutputCache("theCache");
by one o r m ore o f y o u r services to c o n fig u re c o m m o n ly buiIder.AddSqlServerDbContext<BookContext>(" theDb");

54 A s p ir a tio n a l .NET: W h a t Is .N E T A s p ire ? codem ag.com

B e h in d th is s m a ll l i t t l e A d d S e r v ic e D e fa u lts m e th o d is a ll y o u r apps can use. A lo t o f th e s e are used by th e
a lo t o f code. I e nc ou ra ge y o u to Look a t th e code i t d as hb oa rd in s id e th e A p p H o s t. Let's ta k e a lo o k a t th e
g en era te s, as i t c o n fig u re s a lo t o f u s e fu l service s t h a t d as hb oa rd n e x t.

The Application Dashboard

When y o u run a .NET A spire a p p lic a tio n , i t Launches a
dashboard t h a t y o u can use to m o n ito r th e a p p lic a tio n s
in y o u r p ro je c t. For exam ple, F ig u re 6 shows th e i n it ia l
dashboard.

From th is view , y o u can Launch th e e n d p o in ts , and v ie w

th e e n v iro n m e n ts and logs to see how each service is
b e in g Launched. In a d d itio n , y o u can use th e M o n ito rin g
o p tio n s on th e Left to v ie w trace s and m etrics ( t h a t were
d e fin e d in th e S ervice D efau lts ) to see how th e a p p lic a ­
F ig u re 5 : .NET A spire p ro je c ts tio n is d o in g in re al tim e , as seen in F ig u re 7 .

F ig u re 6 : The dashboard

F ig u re 7 : Tracing th e d is trib u te d a p p lic a tio n

codem ag.com A s p ir a tio n a l .NET: W h a t Is .N ET A s p ire ? 55

 Metrics
th<*An<
M o m to n rw

C o n s o le L o a :

r r y .ln T tr u m * 1

n ip

‘■h Ufa-O O I ’ M u c I’ M MM ) PM

F ilte r s

M e tric s o f y o u r d is t r ib u t e d a p p lic a tio n

F ig u r e 8 : kjihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA

Y ou ca n e v e n see p e rfo rm a n c e m e tric s in F ig u r e 8 . e n t it y . W h e n y o u t h i n k a b o u t a m ic ro s e rv ic e im p le m e n t a ­

t i o n , o n e o f t h e d i f f i c u l t t h in g s is t o h a n d le c o n n e c tin g
N o w y o u 'v e se e n h o w t h e t o o lin g a n d t h e d a s h b o a rd w o rk , t h e d if f e r e n t s e rv ic e s . To d o t h is , M ic r o s o ft A s p ire c re a te s
b u t I t h i n k i t 's im p o r t a n t t o u n d e rs ta n d h o w .NET A s p ire s o m e th in g c a lle d an A p p H o s t . T h is a p p h o s t is r e s p o n ­
ca n m a n a g e e a c h p a r t o f y o u r d is t r ib u t e d a p p lic a tio n s . s ib le f o r o r c h e s tr a tin g y o u r a p p lic a tio n . As y o u c an see in
L e t's see t h a t n e x t. F ig u r e 9 , t h e A p p H o s t re p re s e n ts a c o n t e x t a ro u n d w h ic h
t h e s e rv ic e s a re h o u s e d .
Orchestration
T he jo b o f o r c h e s tr a tio n is t o t h i n k o f y o u r a p p lic a tio n as Do n o t c o n fu s e t h e c o n t e x t o f t h e A p p H o s t as a c o n t a in e r
a s in g le s la te o f s e rv ic e s . T h is m e a n s b e in g a b le t o d e p lo y o r w ra p p e r t h a t h o ld s a l l o f t h e c o m p o n e n ts to g e th e r , b u t
an e n t ir e s e t o f s e rv ic e s a n d d e p e n d e n c ie s as a s in g le in s te a d as a c o n d u c to r o f an o r c h e s tr a . I t 's r e s p o n s ib le

AppHost

F ig u r e 9 : T h e A p p H o s t's r e la t io n s h ip t o t h e o t h e r s e rv ic e s

56 A s p ir a tio n a l .NET: W h a t Is .N E T A s p ire ? codem ag.com

f o r s t a r t in g , c o n f ig u r in g , a n d c o n n e c tin g t h e d iffe r e n c e C o m p o nent Type M e t h o d t o A d d t h e S e r v ic e
s e rv ic e s to g e th e r .
D ocker c o n ta in e r A d d D o c k e r()

M ongoD b se rv e r A d d M o n g o D b ()
How Orchestration Works
I f y o u 'v e u s e d ASP.NET C ore, y o u 'd u s u a lly c re a te an a p ­ M ySql se rv e r A d dM yS q l()

p lic a t io n b y u s in g W e b A p p l i c a t i o n . C r e a t e B u i l d e r ( ) . F or P ostgres se rv e r A d d P o s tg re s ()
A s p ire , y o u ca n d o t h is in a s im ila r fa s h io n : SQL se rv e r AddSqLServer()

Redis A d d R e d is C o n ta in e r()
/ / AppHost C# p ro je c t A d d P ro je c t()
var builder =
R abbitM Q A d d R a b b itM Q ()
Di s t r i butedApplicati on.CreateBuiIder(args);
T a b le 1 : C o m m o n c o m p o n e n ts

In s te a d o f w ir in g u p s e rv ic e s a n d a p ip e lin e , y o u a d d t h e
p a rts o f y o u r a p p lic a tio n t h a t y o u n e e d . F or e x a m p le :
C o m p o nent Type M e th o d t o A d d t h e S e r v ic e N u G et Package

buiIder.AddRedisContainer("theCache"); B lob s to ra g e A d dA zureB LobS ervice() A s p ire . A zu re .S to ra g e. B lobs

K eyV ault A d d A z u re K e y V a u ltS e c re ts () A s p ire .M ic ro s o ft.A z u re .C o s m o s
b u iId e r.AddProject<AddressBook_Api> ("theapi"); Table s to ra g e A d dA zu re T ab leS e rv ice () A s p ire . A z u re .S e c u rity . K eyV ault
buiIder.AddProj ect<AddressBook_Blazor>( "fro n te n d ");
S ervice bus A d d A zu re S e rv ic e B u s() Aspire.Azure.Messaging.ServiceBus
CosmosDB A ddA zureC osm osD B () A s p ire . A zure. D ata.Tables
builder
,B uild() T a b le 2 : A z u re c o m p o n e n ts
.Run();

T h is is an a d d it io n a l p r o je c t t h a t 's ru n t o o r c h e s tr a te t h e N o te t h a t t h e c a c h e uses t h e n a m e t h a t w a s d e fin e d w h e n Source Code

e n t ir e a p p lic a tio n . I n t h is e x a m p le , y o u 'r e o r c h e s tr a tin g y o u a d d e d R e d is . I n t h is w a y , t h e c o n n e c tio n t o t h e R edis
t h e th r e e c o m p o n e n ts ( o r s e rv ic e s ) to g e th e r . T he ty p e s i s n 't n e e d e d in t h e A P I p r o je c t. T h is is w h a t is m e a n t The source code can be
o f c o m p o n e n ts t h a t A s p ire s u p p o r ts is q u it e v a r ie d . T h is b y S e rv ic e D is c o v e ry . I t w o rk s id e n t ic a lly f o r B la z o r a p p s , downloaded at https://github.
d o e s n 't m e a n t h a t y o u a re L im ite d t o j u s t th o s e c o m p o ­ t o o . T h is m e a n s t h a t y o u h a v e t o d e a l w it h s h a re d c o n ­ com/shawnwildermuth/
n e n ts , e ith e r . I n t h e p re v ie w , th e r e are a l o t o f s u p p o r te d f ig u r a t io n le ss o f t e n . F or p r o je c t ty p e s t h a t d o n 't h a v e
aspiringDotNet
c o m p o n e n ts . M a n y o f th e s e in c lu d e s im p lif ie d h o s tin g o r n a tiv e s u p p o r t f o r A s p ire , y o u c an s t i l l c o o r d in a te th e m
A z u re h o s tin g , b u t M ic r o s o ft is w o r k in g w it h te a m s a c ro s s th r o u g h e n v ir o n m e n t v a r ia b le s :
c lo u d p ro v id e rs ( i. e . , G o o g le a n d AW S) t o a d d c o m p o n e n ts
f o r th o s e s e rv ic e s t o o . builder.AddNpmApp("frontEnd",
". ,/addressbookapp",
As o f t h e p re v ie w , y o u ca n a lre a d y use s o m e o f t h e m o s t "dev")
c o m m o n c o m p o n e n ts . A s a m p lin g o f th e s e c an be seen .WithReference(api);
in T a b le 1 .
T he c a ll W i t h . R e f e r e n c e s h a re s t h e a p i s e rv ic e 's URL t o
B e ca u se .NET A s p ire ca n a ls o be u s e d t o d e p lo y t o A z u re , t h e a p p u s in g e n v ir o n m e n t v a r ia b le s . F or e x a m p le , t h is
th e r e a re c o m p o n e n ts s p e c ific t o A z u re re s o u rc e s . S om e o f Vue app c an use t h e e n v ir o n m e n t v a r ia b le (a s s u m in g
th e s e c o m m o n A z u re c o m p o n e n ts ca n be s e e n in T a b le 2 . y o u 'r e g o in g t o ru n t h i s as a s e rv e r-s id e n o d e p r o je c t) :

N o w y o u k n o w h o w t o c o m p o s e t h e d is t r ib u t e d a p p b y / / h ttp s ://v ite js .d e v /c o n fig /

u s in g s e p a ra te c o m p o n e n ts , b u t h o w d o y o u h a n d le c o n ­ export default defineConfig({
n e c tin g t h e ap p s ?
define: {
Service Discovery "process.env.APPJJRL":
I n t h e e a r lie r e x a m p le , y o u c an j u s t a d d a R e dis c o n t a in e r J SON. s t r in g if y (
in s e t t in g u p t h e b u ild e r . W h a t d o e s t h a t m e a n ? W he n process. env [ "servi c e s _ th e A p i_ l" ]
t h is a p p lic a tio n is r u n , i t d e p lo y s a s ta n d a rd R e dis d o c k e r
c o n t a in e r a n d c an c o n n e c t i t t o y o u r p r o je c t. F or e x a m p le ,
w h e n y o u a d d t h e A P I p r o je c t, y o u c o u ld s e n d i t a r e fe r ­
e n c e t o t h e R e dis c o n ta in e r : })

/ / AppHost
buiIder.AddRedisContainer("theCache"); W he re A re We?
To b e s u re , th e s e are e a r ly d a y s in t h e life c y c le o f .NET
buiIder.AddProj ect<AddressBook_Api >("theApi") A s p ire . I'm im p re s s e d . In s te a d o f a lw a y s n e e d in g t o d ro p
.WithReference(cache) d o w n in to m u lt ip le c o n ta in e r s a n d m a y b e e v e n K u b e r-
n e te s , .NET n o w has an o r c h e s tr a tio n e n g in e f o r i t s o w n
T h is a llo w s t h e o t h e r a p p lic a tio n s t o use t h e c a c h e u s in g d is t r ib u t e d a p p lic a tio n s . I'm s in c e r e ly c u rio u s t o see w h a t
s im ila r c a lls in t h e p r o je c t s e tu p : t h is lo o k s lik e w h e n i t s h ip s .

/ / API Project S h a w n W ild e r m u th

buiIder.AddRedisOutputCache( " theCache");

codem ag.com A s p ir a tio n a l .NET: W h a t Is .N ET A s p ire ? 57

 O N LIN E Q U IC K ID 2403081

D is trib u te d Caching:
E nhancing S calability and
P erform ance in ASP.NET 8 Core
In the realm of enterprise applications, a deep understanding of distributed caching intricacies and adherence to best practices
are essential to building enterprise applications that are scalable and high performant. You can choose from the popular
distributed caching frameworks, such as Memcache, NCache, Redis, and Hazelcast. In this article, I'll discuss distributed caching,

its b e n e f it s and use cases, a n d th e n te a c h y o u h o w to I f y o u d o n ' t a lr e a d y h a v e V is u a l S t u d io 2 0 2 2 i n s t a l l e d o n

i m p l e m e n t i t i n A S P .N E T C o re . I f y o u 'r e t o w o r k w i t h t h e y o u r c o m p u t e r , y o u c a n d o w n lo a d i t f r o m h e re : h t t p s : / /
c o d e e x a m p le s d is c u s s e d i n t h i s a r t i c l e , y o u n e e d t h e f o l ­ v is u a ls t u d io . m ic r o s o f t . c o m / d o w n lo a d s /.
lo w in g in s t a lle d in y o u r s y s te m :
In t h i s a r t i c l e , I ' l l e x a m in e d i s t r i b u t e d c a c h in g , i t s f e a ­
• V is u a l S t u d io 2 0 2 2 tu re s and b e n e f it s , th e c h a lle n g e s , and how you can
• .N E T 8 . 0 im p le m e n t d is t r ib u t e d c a c h in g u s in g N C ache a n d M em -
• A S P .N E T 8 . 0 R u n t im e C ached.

J o y d ip K a n jila l
j o y d i p ka n j i La L@ y a h o o . c o m

J o y d ip K a n jila l is a n MVP
( 2 0 0 7 - 2 0 1 2 ) , s o ft w a r e
a r c h it e c t , a u t h o r , a n d
s p e a k e r w i t h m o re t h a n
2 0 y e a rs o f e x p e rie n c e .
He h a s m o re t h a n 1 6 y e a rs Cache S erver
o f e x p e r ie n c e in M ic r o s o f t
.N E T a n d i t s r e la te d
t e c h n o lo g ie s . J o y d ip ha s
a u th o re d e ig h t b o o k s ,
C ache-hit T he application reads
m o re t h a n 5 0 0 a r t ic le s ,
data from the cache The application stores/updates data
a n d h a s re v ie w e d m o re into the cache
th a n a d o ze n b o o ks.

A pplication S erver

Cache-m iss. The application reads the

req■uested data from the database

Database

F i g u r e 1 : C a c h in g a t w o r k

58 Distributed Caching: Enhancing Scalability and Performance in ASP.NET 8 Core codem ag.com
 D istnbuted C ache S erver
♦ jihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA

D atabase Server

F i g u r e 2 : D i s t r i b u t e d c a c h in g

What Is Caching? Why Is It Needed? s ie n t s to ra g e , i. e . , t h e m e m o ry fo r fa s te r acce ss t o th e

C a c h in g is a p r o v e n t e c h n i q u e u s e d t o c o l l e c t f r e q u e n t l y cached d a ta f o r a ll s u b s e q u e n t re q u e s ts . I n d is t r ib u t e d
u s e d d a ta in a t r a n s ie n t d a ta s to r e , k n o w n as a c a c h e , f o r c a c h in g , d a t a is s t o r e d o n m u l t i p l e n o d e s , i . e . , t h e c a c h e d
s u b s e q u e n t r e tr ie v a l w h e n re q u e s te d b y th e a p p lic a tio n . d a t a is s p r e a d a c r o s s s e v e r a l s y s t e m s . I n c l i e n t - s i d e c a c h ­
T h is d a t a s t o r e c a n r e s id e i n a t r a n s i e n t s t o r a g e s u c h as in g , th e cached d a t a is s t o r e d on th e c lie n t c o m p u te r,
m e m o ry , o r in a p e r m a n e n t s to r a g e s u c h as in a file , o r s u c h as o n th e w e b b ro w s e r.
e v e n in s id e a d a t a b a s e .

I t s h o u ld b e n o t e d t h a t w h e n a r e q u e s t f o r d a t a is m a d e , An Overview of Distributed Caching

t h e c a c h in g s y s t e m f ir s t a tte m p ts to d e t e r m in e w h e t h e r An e x t e n s io n of c a c h in g c a lle d d is t r ib u t e d c a c h in g is
t h e r e q u e s t e d d a t a a lr e a d y e x is t s i n t h e c a c h e b e f o r e a t ­ th e p ro c e s s o f d is t r ib u t in g th e c a c h e a c ro s s s e v e ra l s e rv ­
te m p tin g to r e t r ie v e it. I f th e d a ta is a v a ila b le in th e e rs o r m a c h in e s . In a d i s t r i b u t e d c a c h in g e n v i r o n m e n t ,
c a c h e , y o u s h o u ld b e a b le t o g e t i t f r o m t h e r e r a t h e r t h a n th e cache s e rv e rs a re s p re a d a c ro s s m u lt ip le nodes or
a c c e s s in g t h e d a t a f r o m t h e o r i g i n a l s o u r c e . A m a jo r b e n ­ m a c h in e s , a l l o w i n g f o r s c a la b ility a n d im p r o v e d p e rfo r­
e f i t o f t h i s is t h a t d a t a c a n b e r e t r ie v e d f r o m f a s t e r c a c h e m ance. The m a in p u rp o s e o f d is t r ib u t e d c a c h in g is to
s t o r a g e s i n a s i g n i f i c a n t l y s h o r t e r p e r io d o f t i m e i n s t e a d s to re f r e q u e n tly accessed d a ta in m e m o ry , ra th e r th a n
o f s lo w e r s t o r a g e d e v ic e s , s u c h a s r e m o t e d a ta b a s e s o r f e t c h i n g i t f r o m t h e h a r d d r iv e o r a r e m o t e d a t a b a s e . T h is
h a r d d r iv e s . (S e e F i g u r e 1 . ) im p r o v e s t h e s p e e d o f d a t a a c c e s s b e c a u s e r e t r i e v i n g d a t a
fro m m e m o r y is f a s t e r t h a n d is k a c c e s s . (S e e F i g u r e 2 . )
C a c h in g s t r a t e g ie s need to c o n s id e r t r a d e - o f f s b e tw e e n
cache s iz e , cache e v ic tio n p o l ic ie s , and d a ta c o n s is ­ D i s t r i b u t e d c a c h e s y s t e m s c o m b in e t h e m e m o r y o f s e v e r a l
t e n c y . O p t im a l p e r f o r m a n c e r e q u ir e s a b a la n c e b e tw e e n n e t w o r k e d c o m p u t e r s i n t o a s i n g le m e m o r y s t o r e f o r f a s t
cache u tiliz a t io n , d a ta fre s h n e s s , a n d d a ta access p a t­ d a t a a c c e s s . T h is e x p la in s w h y d i s t r i b u t e d c a c h e s a r e n 't
t e r n s . T h e r e a r e s e v e r a l c a c h in g t y p e s , s u c h a s i n - m e m o r y c o n s t r a in e d b y t h e l i m i t o f a v a ila b le m e m o r y o f a s in g le
c a c h in g , d i s t r i b u t e d c a c h in g , a n d c l i e n t - s i d e c a c h in g . I n c o m p u te r o r h a rd w a re c o m p o n e n t. T h e y m a y g o b e y o n d
c a s e o f i n - m e m o r y c a c h in g , t h e d a t a is s t o r e d i n a tra n ­ th e s e b o u n d a r ie s by c o n n e c t in g n u m e ro u s c o m p u te rs ,

codem ag.com Distributed Caching: Enhancing Scalability and Performance in ASP.NET 8 Core 59
 fo rm in g a d is trib u te d a rc h ite c tu re o r d is trib u te d clus te r, fre q u e n tly accessed clo se r to th e a p p lic a tio n .
e n a b lin g increased pro cessing p ow er and sto ra ge capac­ • R e lia b ilit y : D is trib u te d c a chin g prom otes r e lia b ility
ity . (See jihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA
F ig u re 3 .) by s to rin g d ata across m u ltip le servers. I f a cache
server fa ils , th e re m a in in g servers can s t i l l serve th e
N ote t h a t a d is trib u te d cache is based on DHT, an ac­ cached d a ta , e n s u rin g data a v a ila b ility .
ronym fo r D is trib u te d Hash Table. In th is DHT, th e data • H ig h a v a ila b ilit y : D is trib u te d ca c h in g system s can
is sto re d as ke y-v alue pa irs. Each p a rtic ip a tin g node can e a s ily ha nd le increase d tr a f fic by a d d in g servers,
re trie v e th e va lu e using a key. DHT a llo w s d is trib u te d e n a b lin g th e system s to scale d y n a m ic a lly w ith o u t
caches to be scaled d y n a m ic a lly by c o n tin u o u s ly m anag­ d is ru p tin g th e e x is tin g o p e ra tio n s , hence en su rin g
in g a d d itio n s , d e le tio n s , and node fa ilu re s . h ig h a v a ila b ility o f th e cached d ata .
• R e s ilie n c e : C aching prov ides enhance d res ilie n ce
Some ty p ic a l exam ples o f w id e ly used d is trib u te d ca ch in g a g a in s t te m p o ra ry t r a f fic spikes o r database o u t­
system s in c lu d e NCache, Redis, and M em cached. These ages. The cache da ta can be accessed q u ic k ly , p ro ­
system s leverage d is trib u te d cache m echanism s and o ffe r v id in g a s m o o th e r user experience.
APIs fo r fa s te r da ta sto ra ge and re trie v a l. CBA

B e n e f its
B u ild in g a S im ple D is trib u te d
D is trib u te d ca c h in g o ffe rs se veral b e n e fits , in c lu d in g : A p p lic a tio n in ASP.NET Core
It's tim e fo r w r itin g som e code. L e ts now exam ine how to
• S c a la b ility : W ith a surge in tr a f fic fo r an a p p lic a ­ b u ild a sim p le ASP.NET Core 7 Web A P I a p p lic a tio n using
tio n , y o u can add a d d itio n a l cache servers to th e GraphQL.
d is trib u te d cache system w ith o u t d is ru p tin g e x is t­
in g o p e ra tio n s . C r e a te a N e w A S P . N E T C o re 8 P r o je c t
• R ed u ced la te n c y : W ith d is trib u te d c a c h in g , cached in V is u a l S t u d io 2 0 2 2
data is u su a lly sto re d in an in -m e m o ry data sto re , You can create a p ro je c t in V is u al S tu d io 2022 in several
e n a b lin g b la z in g fa s t access and re d uc in g la te n c y ways. When yo u launch V is u al S tu d io 20 2 2, y o u 'll see th e
caused by fe tc h in g da ta from th e d is k o r any rem ote S ta rt w in d ow . You can choose "C o n tin u e w ith o u t code"
services. to Launch th e m ain screen o f th e V is ua l S tu d io 2022 IDE.
• P e rfo rm a n c e : A d is trib u te d c a chin g system reduces
th e Latency ass ocia ted w ith fe tc h in g fre q u e n tly ac­ To create a new ASP.NET Core 8 P ro je c t in V is u al S tud io
cessed d ata fro m disks o r re m ote services by ca ch in g 2022 :

F ig u re 3 : D is trib u te d
ca ch in g w ith m u ltip le
nodes

60 Distributed Caching: Enhancing Scalability and Performance in ASP.NET 8 Core codem ag.com
 1. S ta rt th e V is ua l S tu d io 2022 IDE. data clo se r to users in d iffe r e n t reg io ns . This de­
2. In th e "C reate a new p ro je c t" w in d ow , s e le c t "ASP. creases th e a m o u n t o f tim e th a t's o th e rw is e needed
NET Core Web A P I" and c lic k N ext to m ove on . to access data across Long n e tw o rk distan ces , im ­
3. S p e cify th e p ro je c t nam e as D is trib uted C ac h in g D e m o p ro v in g perfo rm an ce and re du cing laten cy .
and th e p a th w here i t sh ou ld be crea ted in th e "C o n­
fig u re y o u r new p ro je c t" w in d ow . I f y o u w a n t th e
s o lu tio n f ile and p ro je c t to be cre ate d in th e same D is trib u te d C aching Best Practices
d ire c to ry , y o u can o p tio n a lly check th e "P la ce s o lu ­ Im p le m e n tin g d is trib u te d c a chin g requires adherence to
tio n and p ro je c t in th e same d ire c to ry " checkbox. several b e s t p ra ctices :
C lick N ext to m ove on .
4. In th e n e x t screen, s p e c ify th e ta rg e t fra m e w o rk and • Cache in v a lid a t io n : Decide on a cache in v a lid a ­
a u th e n tic a tio n ty p e as w e ll. Ensure t h a t th e "C on­ tio n s tra te g y e n s u rin g t h a t th e cached d ata re­
fig u re fo r HTTPS," "E nable Docker S u p p o rt," and th e m ains up to da te . C onsider o p tio n s Like tim e - to - liv e
"E na ble O penAP I s u p p o rt" checkboxes are unchecked (TTL) based e x p ira tio n , e x p lic it cache in v a lid a tio n ,
because y o u w o n 't use any o f th es e in th is exam ple. o r e v e n t-d riv e n in v a lid a tio n m echanism s based on
5. Because you w o n 't be using m in im a l APIs in th is ex­ data changes o r updates.
am ple , rem em ber to leave th e U se c o n tr o lle rs ( u n ­ • E v ic tio n p o lic ie s : Choose a p p ro p ria te e v ic tio n
c h e c k t o use m in im a l A P Is ) checkbox checked. p o lic ie s fo r y o u r d is trib u te d cache. Cache e v ic tio n
6. C lick Create to c o m p le te th e process. p o lic ie s help m anage cache d ata and d e te rm in e how
and w hen th e y 're rem oved fro m th e cache. There are
I ' l l use th is p ro je c t in th e su bs e qu e nt se ctio n s o f th is th re e ty p e s o f e v ic tio n p o lic ie s : le a s t re c e n tly used
a rtic le . (LR U ), Least fre q u e n tly used (LFU), and fix e d size
e v ic tio n s .
• Cache s iz in g a n d c a p a c ity p la n n in g : E stim ate th e
D is trib u te d C aching Use Cases cache size and c a p a c ity re q uire m e nts based on th e
There are several use cases o f d is trib u te d c a chin g across v o lu m e o f d ata and ex pected w o rk lo a d . C onsider
d iffe r e n t dom ains and in d u s trie s . Here are a fe w com m on fa c to rs Like m em ory a v a ila b ility , cache server capa­
scenarios w here d is trib u te d ca chin g can help: b ilitie s , and fu tu re g ro w th to ensure t h a t th e cache
can ha nd le th e a n tic ip a te d load.
• H ig h - t r a f f ic w e b s ite s : D is trib u te d ca c h in g can be • P ro p e r k e y d e s ig n : Design cache keys w ith care.
used to sto re fre q u e n tly accessed w eb pages, HTML Use m e a n in g fu l and u n iq u e keys t h a t are easy to
fra g m e n ts , o r s ta tic c o n te n t. By ca chin g th es e e l­ ge nerate and parse. A vo id o ve ru sin g co m p le x o b ­
em ents on d is trib u te d cache servers clos e r to th e je c ts o r d yn am ic data as keys, as i t can n e g a tiv e ly
users, th e y can be served q u ic k ly and reduce th e im p a c t cache perfo rm an ce.
load on ba ck -e nd servers, im p ro v in g w e b s ite p e rfo r­ • M o n ito r a n d t u n e p e r fo rm a n c e : R eg ularly m o n ito r
m ance and s c a la b ility . and m easure th e perfo rm an ce o f y o u r d is trib u te d
• D a ta b a s e c a c h in g : D is trib u te d ca chin g can be em ­ cache to id e n tify b o ttle n e c k s , cache h it ra tio s , and
p lo ye d to cache fre q u e n tly accessed database que ­ review o th e r pe rfo rm an ce in d ic a to rs . A d ju s t cache
ries o r re s u lt sets. By s to rin g th is in fo rm a tio n in c o n fig u ra tio n , e v ic tio n p o lic ie s , o r cache server ca­
m em ory, you can q u ic k ly respond to requests a v o id ­ p a c ity , i f necessary, to o p tim iz e perform ance.
in g th e need fo r p o s s ib ly expensive database queries • Im p le m e n t f a ilo v e r a n d r e p lic a tio n : Ensure high
o r re d u n d a n t database h its . a v a ila b ility and f a u lt to le ra n c e by im p le m e n tin g
• M ic ro s e rvic e s a r c h it e c tu r e : In a ty p ic a l m ic ro se r­ fa ilo v e r m echanism s and data re p lic a tio n across
vices a rc h ite c tu re , th e re can be m any d iffe r e n t ser­ cache servers in case o f fa ilu re s . This helps m a in ­
vices c o m m u n ic a tin g w ith each o th e r, w h ic h can ta in c o n s is te n c y and ensures u n in te rru p te d access
increase la te n c y due to n e tw o rk ro u n d trip s . By us­ to cached da ta .
in g d is trib u te d c a c h in g , fre q u e n tly accessed data • S e c u r ity c o n s id e ra tio n s : Pay a tte n tio n to s e c u rity
o r c o m p u te d re su lts can be cached, m in im iz in g th e aspects, such as access c o n tro l, a u th e n tic a tio n , and
need fo r re pea ted re m ote calls and re du cing Latency. e n c ry p tio n , to p ro te c t s e n s itiv e da ta sto re d in th e
• C o n te n t d e liv e r y n e tw o rk s (C D N s ): CDNs Leverage d is trib u te d cache.
d is trib u te d c a chin g to p e rs is t and also serve s ta tic • A p p r o p r ia te d a ta s e le c tio n : N ot a ll d ata is s u ita b le
c o n te n t t h a t in clu d e s im ages, J a v a S c rip t, and CSS fo r c a c h in g . Id e n tify d ata th a t's fre q u e n tly read b u t
file s . By ca c h in g c o n te n t in m u ltip le edge servers in fre q u e n tly u pd a te d , as th is ty p e o f da ta b e n e fits
lo c a te d g e o g ra p h ic a lly clos e r to users, CDNs can de­ m os t fro m c a c h in g . A vo id c a chin g d ata t h a t changes
liv e r c o n te n t fa ste r, re d uc in g la te n c y and im p ro v in g very fre q u e n tly .
th e user experience. • Cache in v a lid a t io n s tr a te g y : Im p le m e n t a ro b u s t
• R e a l- tim e a n a ly tic s : In d a ta -in te n s iv e a p p lic a ­ cache in v a lid a tio n s tra te g y to ensure d ata c o ns is ­
tio n s , d is trib u te d c a chin g can be used to sto re pre- te n cy. T his co u ld be tim e -b a s e d (e .g ., TTL— tim e to
processed o r c o m p u te d d ata t h a t is fre q u e n tly ac­ liv e ), e v e n t-d riv e n , o r a c o m b in a tio n o f b o th . When
cessed fo r re a l-tim e a n a ly tic s . By c a chin g th is d ata , a change occurs in th e u n d e rly in g d a ta , it's re com ­
a p p lic a tio n s can re trie v e in s ig h ts q u ic k ly , a v o id ­ m ended t h a t y o u up d ate th e cache to ensure t h a t
in g th e Latency associa ted w ith rep rocessing v a s t y o u r cached data is alw ays in sync w ith th e data
am o un ts o f raw d ata . re s id in g in th e da ta store .
• G e o g r a p h ic a lly d is tr ib u t e d a p p lic a tio n s : In g lo b ­ • D a ta p a r t it io n in g a n d s h a rd in g : D is trib u te data
a lly d is trib u te d a p p lic a tio n s , d is trib u te d ca ch in g across d iffe r e n t cache servers to balance th e Load
can be used to sto re and serve fre q u e n tly accessed and reduce risks o f a s in g le p o in t o f fa ilu re . Use

codem ag.com Distributed Caching: Enhancing Scalability and Performance in ASP.NET 8 Core 61
 sharding techniques to partition data effectively • Load balancing: Employ Load balancing to distrib­
based on usage patterns or other relevant criteria. ute requests evenly across cache servers, thus en­
• Handling cache misses: Design your system to suring use of resources.
handle cache misses gracefully. When the data re­ • Scalability: Ensure that your caching solution can
quested by the application isn't available in the scale horizontally. As the demand increases, you
cache, it's known as a cache miss. Optimizing this should be able to add more cache servers to the
fallback mechanism is essential for maintaining per­ system without significant changes to the existing
formance.kjihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA infrastructure.

T he app lica tio n s e a rc h e s fo r

C a c h e S erver* s
data m the ca c h e initially.

m e a p p lic a tio n upd a te s th e ca cn e witn

th e data retn ev e d from the da taba se

A p p lic a tio n

i f th e ap p lic a tio n is u n a b le to fin d

the re q u e s te d data in th e cache
it se arche s for it in the da taba se

D a ta b a s e

Figure 4: The cache-aside pattern

Figure 5: The read-through pattern

A p p lic a tio n
C a c h e S e rv e r
Figure 6: Data is served from the cache for subsequent requests

62 Distributed Caching: Enhancing Scalability and Performance in ASP.NET 8 Core codem ag.com
 • Data security: You should establish the necessary • When a request arrives, the application searches for
security measures by implementing security mea­ the data in the cache.
sures that include encryption of data in transit and • If the data is available, the application returns the
at rest, proper access controls, and authentication cached data. This phenomenon is also known as a
mechanisms, etc. cache-hit.
• Monitoring and analytics: You should monitor the • On the other hand, if the data that has been re­
performance of your cache system on a regular ba­ quested is unavailable in the cache, the application
sis. In this regard, you can take advantage of met­ retrieves data from the database and then populates
rics such as cache hit, miss rates, and load patterns the cache with this data.
to optimize your caching strategy.
• Data synchronization: Ensure that the cache is There are a few benefits of this approach:
synchronized across all nodes in distributed envi­
ronments. To maintain data integrity, you can take • The cache contains only the data the application
advantage of techniques such as distributed Locking has requested, thus ensuring that the cache size is
or atomic operations. optimal and cost-effective.
• Selecting the right tool: Selection of the right • This strategy is simple and can provide immediate
caching solution that satisfies the requirements performance benefits.
of your application is extremely important. Redis,
Memcached, and Hazelcast are popular tools with The downsides to this approach include:
varying features and capabilities.
• Avoid cache stampede: Implement strategies Like • Because the data is loaded into the cache only in
staggered TTLs, pre-computation, or using a proba­ the event of a cache miss, the in itia l response time
bilistic early expiration to avoid a cache stampede, increases because of the additional roundtrips re­
where multiple instances try to compute and store quired to the database and the cache.
the same data in the cache simultaneously. • Using this approach, the data is stored directly in
the database, which may result in discrepancies be­
tween the database and the cache.
Distributed Caching Patterns
Cache patterns are an approach to application design Another pattern is called read through (see Figure 5). In
that emphasizes caching techniques to improve seal- this pattern, the application queries the cache first. The
ability, performance, and responsiveness. These patterns cache then interacts with the underlying database on a
provide valuable insights and suggestions for implement­ lazy Load basis. If the data requested by the application is
ing caching techniques. Typically, these patterns are used unavailable in the cache, it's called a cache miss. In this
with distributed caching systems that include NCache or case, the application retrieves the requested data from
Memcached. You can take advantage of these patterns to the data store (i.e., a database or a service, etc.), stores
reduce Latency and improve your application's workload this data into the cache, and then returns it.
processing capabilities.
The cached copy of the same data is then served for all
One such pattern is called cache aside (see Figure 4). subsequent requests for the same piece of data.
The cache aside pattern adopts a lazy loading approach
and is the most used caching strategy. Here's how this Another pattern is write through (see Figure 7): This
pattern works: pattern is similar to read through, but instead of read-

a p p iic a u o r i
D a ta b a s e C a c h e S e rv e r

Figure 7: The write-through pattern

a p p lie d nun
D a ta b a s e C a c h e S e rv e r

Figure 8: The write-around pattern

codem ag.com Distributed Caching: Enhancing Scalability and Performance in ASP.NET 8 Core 63
 T he d a ta s to re is u p d a te d w ith th e c a c h e d
da ta in fre q u e n tly , i.e.. o n c e in a w h ile

A p p lic a tio n w rite s o n ly to th e

c a c h e in lieu o f th e d a ta b a s e

Database Cache S erver jihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA

F ig u re 9 : The w rite -b e h in d o r w rite -b a c k p a tte rn

in g data from th e ba ck -e nd system , i t w rite s data to th e cache. A cache e n try is rem oved w hen its tim e - to -
cache. In o th e r w ords, w hen using a w rite th ro u g h cache, liv e (TTL) expires. You can e m p lo y th is m eth o d when
th e ord er in w h ic h th e cache is p o p u la te d is reversed. th e d ata in y o u r a p p lic a tio n has a s h o rt d u ra tio n ,
A fte r a cache m iss, th e cache is n o t Lazy Loaded. In s te a d , lik e session data o r refreshed d ata . I t can prove b e n ­
it's p ro a c tiv e ly u pd ated im m e d ia te ly fo llo w in g an up da te e fic ia l in scenarios w here y o u r a p p lic a tio n relie s on
to th e database. U sing th is approach makes sense w hen data t h a t has a s h o rt Lifespan such as session data
Using NCache in w r itin g o p e ra tio n s are fre q u e n t and a p p lic a tio n p e rfo r­ o r data t h a t undergoes fre q u e n t updates.
ASP.NET Core m ance is p ara m o u n t.
D is t r i b u t e d C a c h in g C h a lle n g e s
To work with NCache in
A n o th e r p a tte rn is w r it e a ro u n d (see F ig u re 8 ) : In th e A lth o u g h th e re are b e n e fits , d is trib u te d c a chin g poses
ASP.NET Core, install the
w rite aro und a pp ro a ch , th e d ata is w ritte n d ire c tly to th e c e rta in cha lle ng es as w e ll:
following NuGet package:
data sto re w ith o u t s to rin g i t in th e cache. When th e ap ­
NCache.Microsoft.Extensions. p lic a tio n perfo rm s a read o p e ra tio n , th e d ata is placed in • D a ta c o n s is te n c y : The m ain c h alle n g e w ith d is ­
Caching.OpenSource. th e cache. T his s tra te g y is a good ch oice i f y o u r a p p lic a ­ tr ib u te d ca chin g is m a in ta in in g da ta con sis te ncy .
tio n d o e s n 't need to reread fre q u e n tly w ritte n d ata . Caches m ay have d iffe r e n t da ta ve rsion s o r m ay ex­
pe rience re p lic a tio n delays, le a d in g to d ata in c o n ­
A n o th e r p a tte rn is w r it e b e h in d o r w r it e b ac k (see F ig ­ sis tency.
u re 9 ) : In th e w rite back ap proa ch , data is w ritte n d i­ • Cache in v a lid a t io n : Im p le m e n tin g p ro p e r cache in ­
re c tly to th e cache alo n e in s te a d o f th e data sto re . The v a lid a tio n and s y n c h ro n iz a tio n m echanism s o r using
data sto re is u pd ated w ith th is cached data a fte r s p e c ific e v e n tu a l c o n s is te n c y m odels can help m itig a te th is
in te rv a ls o f tim e , o r based on c e rta in p re -d e fin e d c o n d i­ ch alle n g e. In v a lid a tin g caches can be c h a lle n g in g ,
tio n s . A lth o u g h th is approach is b e n e fic ia l in w r ite - in te n ­ e s p e c ia lly w hen data is fre q u e n tly c h a n g in g o r in ­
sive a p p lic a tio n s due to lo w Latency and h ig h th ro u g h p u t te rd e p e n d e n t. E nsuring t h a t th e cache is refreshed,
c a p a b ilitie s , th e re are risks o f d ata loss in th e e v e n t o f a i.e ., u pd ated w hen u n d e rly in g data changes, o r em ­
system crash e tc . CBA p lo y in g s tra te g ie s , such as cache e x p ira tio n based
on tim e , can help m a in ta in cache v a lid ity .
C a c h e E v ic t io n S tr a te g ie s • Cache c o h e re n c e : Cache coherence refers to en­
Cache e v ic tio n in v o lv e s th e re m ov a l o f cache e n trie s o r s u rin g th a t a ll cache servers have c o n s is te n t
data from a cache based on s p e c ific c o n d itio n s o r rules d ata . A c h ie v in g cache coherence can be c h a lle n g ­
b e in g s a tis fie d . In d is trib u te d cache system s, cache e v ic ­ in g , e s p e c ia lly in scenarios w here d ata is u pd ated
tio n a lg o rith m s are used to pro vide space fo r new data fre q u e n tly o r w hen m u ltip le cache servers are in ­
and p re v e n t th e cache from g o in g beyond its ca pa city . v o lv e d . E m ploy ing d is trib u te d cache s o lu tio n s t h a t
There are va rio u s cache e v ic tio n s tra te g ie s a v a ila b le fo r p ro vid e s tro n g c o n s is te n c y guaran tees or using
re m ov a l o f cache e n trie s . Here are th e key cache e v ic tio n cache coherence p ro to c o ls can help address th is
s tra te g ie s y o u can e xp lo re : ch alle ng e.
• Cache w a rm -u p : When a cache is e m p ty o r a fte r
• L e a s t r e c e n tly used (L R U ): Per th is s tra te g y , th e a cache server re s ta rts , i t tak es tim e to p o p u la te
cache ite m th a t's been accessed th e le a s t re c e n tly th e cache w ith fre q u e n tly accessed d ata . T his i n i ­
is rem oved fro m th e cache w hen th e cache reaches t i a l p erio d can re s u lt in cache m isses and increased
its m axim um ca p a city. N ote t h a t th is approach as­ la te n c y u n t il th e cache is w arm ed up. Proper cache
sumes e n trie s t h a t have been th e Least used w ill n o t w a rm -u p s tra te g ie s , such as preLoading c o m m o n ly
be used ag ain in th e near fu tu re . accessed data o r using w a rm -u p s c rip ts , can help
• M o s t r e c e n tly used (M R U ): Cache e n trie s t h a t have m in im iz e th is issue.
been accessed m os t re c e n tly are e v ic te d w hen th e • Cache e v ic tio n s tr a te g ie s : Since da ta access p a t­
cache is f u ll. As per th is a s s u m p tio n , e n trie s t h a t te rn s m ay d iffe r across th e nodes, i t is q u ite c h a l­
have been used m ore re c e n tly are less Likely to be le n g in g to im p le m e n t cache e v ic tio n s tra te g ie s in
used aga in in th e near fu tu re . a d is trib u te d cache e n v iro n m e n t. C areful c o n s id e r­
• L e a s t f r e q u e n t ly used ( L F U ) : T his p o lic y s tip u la te s a tio n sh o u ld be give n to s e le c tin g an e v ic tio n s tr a t­
t h a t w hen th e m axim um cache c a p a c ity is reached, egy t h a t a lig n s w ith y o u r a p p lic a tio n 's d ata access
th e cache e le m e n t accessed Least fre q u e n tly w ith in p a tte rn s and re qu irem e nts.
a c e rta in p e rio d is rem oved. I t asserts t h a t e n trie s • S c a lin g : As th e a p p lic a tio n load and da ta v o lu m e in ­
used less o fte n are u n lik e ly to be accessed in th e crease, s c a lin g th e d is trib u te d cache can becom e a
fu tu re . ch alle n g e. Proper p la n n in g and a rc h ite c tu re sh ou ld
• T im e - t o - liv e (T T L ): T his p o lic y d efin es th e d u ra tio n be in place to ensure t h a t th e cache can ha ndle th e
fo r w h ic h a cache e n try sh o u ld rem ain v a lid in th e added Load e ffe c tiv e ly .

64 Distributed Caching: Enhancing Scalability and Performance in ASP.NET 8 Core codem ag.com
 • Cache s y n c h r o n iz a tio n : In m u lti- le v e l c a chin g sys­ C o n f ig u r in g N C a c h e a s a n I D is t r ib u t e d C a c h e P r o v id e r
tem s, w here m u ltip le caches, Like lo c a l cache and To w ork w ith d is trib u te d c a chin g using NCache, in v o k e
d is trib u te d cache, are used, e n s u rin g p ro p e r sy n ­ th e A ddN C acheD istributedC ache m eth o d in th e Program ,
c h ro n iz a tio n and c o n s is te n c y across d iffe r e n t cache cs f ile to re g is te r NCache w ith th e loC c o n ta in e r. Note
layers can be c h a lle n g in g . You can p re v e n t in c o n ­ t h a t th e A ddN C ach eD istrib uted C ache() m eth o d in NCache
sis te nc ie s in y o u r da ta by im p le m e n tin g th e r ig h t is an e xte n s io n m eth o d o f th e A d d D is trib u te d C a c h e ()
s y n c h ro n iz a tio n s tra te g ie s . m eth o d p e rta in in g to ASP.NET Core.
• N e tw o rk o v e rh e a d : D is trib u te d c a chin g system s re­
q uire c o m m u n ic a tio n and s y n c h ro n iz a tio n betw een builder.Services.AddNCacheDistributedCache
cache servers, w h ic h in tro d u c e s n e tw o rk overhead. (configuration =>
H igh n e tw o rk la te n c y o r Lim ited b a n d w id th can a f­ {
fe c t cache perfo rm an ce and o v e ra ll a p p lic a tio n re­ configuration.CacheName = "demoCache";
sponsiveness. configuration.EnableLogs = true;
• C o m p le x ity a n d m a in te n a n c e : For th e cache in ­ configuration.ExceptionsEnabled = true;
fra s tru c tu re to o pe ra te s m o o th ly , i t m u s t be p ro p ­ });
e rly m a in ta in e d , m o n ito re d , and tro u b le s h o t w ith
p ro p e r e x p e rtis e and resources. Im p le m e n ta tio n
o f a d is trib u te d c a chin g system , in th e a p p lic a ­ I f y o u 'd lik e to w ork w ith m u ltip le caches, use th e fo llo w ­
tio n a rc h ite c tu re in tro d u c e s increased co m p le x ­ in g code s n ip p e t to c o n fig u re th e m in th e Program .es file :
it y in te rm s o f im p le m e n ta tio n , m ana ge m ent, and
m o n ito rin g . builder.Services.
• A p p lic a tio n c o m p a t ib ilit y : Some a p p lic a tio n s m ay AddNCacheDistributedCacheProvider
n o t be designed to w ork seam lessly w ith d is trib u te d (options =>
c a chin g o r m ay have dependencies t h a t d o n 't sup ­ {
p o rt ca c h in g e ffe c tiv e ly . E valuate and m o d ify th e options.Cacheconfigurations =
a p p lic a tio n to ensure c o m p a tib ility w ith th e d is tr ib ­ new NCacheConfiguration[] {
u te d ca chin g ap pro ach . new NCacheConfiguration() {
CacheName = "myFirstCache",
EnableLogs = true,
D is trib u te d C aching in ASP.NET Core ExceptionsEnabled = true
S u p p o rt fo r d is trib u te d ca c h in g is in - b u ilt in ASP.NET },
Core us ing th e ID is trib u te d C a c h e in te rfa c e . The ID is tr ib - new NCacheConfiguration(){
utedC ache in te rfa c e helps y o u to p lu g in any t h ir ty - p a r ty CacheName = "mySecondCache",
c a chin g fram ew orks. L is tin g 1 illu s tra te s how th e ID is ­ EnableLogs = true,
trib u te d C a c h e in te rfa c e looks. ExceptionsEnabled = true }
};
As e v id e n t fro m th e source code o f th e ID is trib u te d C a c h e });
in te rfa c e , its G et m eth o d re tu rn s b y te [ ]. N evertheless, th e
fra m e w o rk prov ides e xte n s io n m e th od s fo r w o rk in g w ith
s trin g o b je c ts . In a d d itio n , y o u can im p le m e n t custom S e t t in g U p N C a c h e in A S P .N E T C o re
e x te n s io n m e th od s to make i t w ork w ith o th e r d ata ty p e s. To se tu p NCache in y o u r system , fo llo w th es e steps:

E n a b lin g D is t r i b u t e d C a c h in g in A S P .N E T C o r e 1. D ow nload th e NCache in s ta lle r to y o u r co m p uter.

You can en able d is trib u te d c a chin g in y o u r ASP.NET Core 2. Open a com m and p ro m p t w in d o w as a d m in is tra to r
a p p lic a tio n by a d d in g th e fo llo w in g code s n ip p e t in th e and run th e m siexec.exe u t ilit y to in s ta ll NCache.
Program .es file :

buiIder.Services.AddDistributedMemoryCacheO; L is tin g 1 : The ID is trib u te d C a c h e In te rfa c e

public interface IDistributedCache

You can now use th e ID is trib u te d C a c h e in te rfa c e to im ­ {
p le m e n t d is trib u te d c a chin g in y o u r ASP.NET Core a p p li­ byte[] Get(string key);
c a tio n . Task<byte[]>
GetAsync(string key);
void Set(string key,
I m p le m e n t in g D is t r i b u t e d C a c h in g U s in g N C a c h e
byte[] value,
NCache is a fa s t, o pe n-so urce , c ro s s -p la tfo rm , d is t r ib u t­ Di stributedCacheEntryOptions
ed, in -m e m o ry c a chin g fram e w o rk . It's a d e p t a t e n h a n c ­ options);
in g th e s c a la b ility and pe rfo rm ance o f y o u r a p p lic a tio n s Task SetAsync(string key,
by c a chin g fre q u e n tly accessed data in s te a d o f s to rin g to byte[] value,
and re trie v in g th e data and from th e database. There are Di s tri butedCacheEntryOpti ons
options);
a v a rie ty o f c a chin g fe a tu re s in NCache, in c lu d in g o b je c t
void Refresh(string key);
c a c h in g , SQL c a c h in g , f u ll- t e x t search, and d is trib u te d Task RefreshAsync(string key);
c a c h in g , and th e re are use cases such as w eb sessions, void Remove(string key);
ASP.NET o u tp u t c a c h in g , and e n te rp ris e c a c h in g . NCache Task RemoveAsync(string key);
s u p p o rts Docker and K ubernetes and va rio u s c a chin g t o ­ }
p o lo g ie s , in c lu d in g re p lic a te d , p a rtitio n e d , and c lie n t­
side ca c h in g .

codem ag.com Distributed Caching: Enhancing Scalability and Performance in ASP.NET 8 Core 65
 S to re /R e trie v e d ata using th e ID is trib u te d C a c h e in te rfa c e
L is tin g 2 : kjihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA T hat's a ll y o u have to do to in s t a ll NCache on y o u r com ­
p uter. CBA
app.MapGet( "/T e st",
async (IDistributedCache cache,
S to r e a n d R e tr ie v e a n O b je c t U s in g N C a c h e
IHttpClientFactory httpClientFactory) =>
{ C onsider th e fo llo w in g class:
const s trin g cacheKey = "Test";
List<Author>? authors = n u ll; public class Author
var cachedData =
{
await cache.GetStringAsync(cacheKey);
public in t Authorld { get; set; }
public string? FirstName { get; set; }
i f ( ! string.IsNuUOrEmpty(cachedData))
{ public string? LastName { get; set; }
authors = public bool IsActive { get; set; }
System.Text.J son.J sonSeri a li ze r. Deserialize }
<List<Author>>(cachedData);
} L is tin g 2 illu s tra te s how y o u can leverage NCache to
sto re and re trie v e an o b je c t:
else
{
Author user = new AuthorQ N ote t h a t fo r th e sake o f s im p lic ity , I'v e used a m in i­
{ m al A P I o n ly to d e m o n s tra te how NCache w orks . I'v e
Authorld = 1, added o n ly one re cord here fo r th e sake o f b re v ity . When
FirstName = "Joydip",
y o u e xec u te th e a p p lic a tio n and brow se th e / t e s t e n d ­
LastName = " K a n jila l" ,
IsActive = true p o in t, th e a u th o r re cord is d is p la y e d in th e w eb brow ser.

}; W hen th is e n d p o in t is h it th e f ir s t tim e , th e a u th o r re ­
cord is s to re d in th e cache. For a ll s u b s e q u e n t req ue sts
authors = new List<Author>(); to th is e n d p o in t, th e d a ta is fe tc h e d fro m th e cache
authors.Add(user); based on th e e x p ira tio n tim e s e t in th e c o n fig u r a tio n .
In th is exa m p le , th e d a ta resides in th e cache fo r 60
await cache.SetStringAsync
seconds.
(cacheKey,
System.Text.J son.J sonSeri a liz e r.S e ria liz e
(authors), The c o m p le te code Listing o f th e Program .es f ile is given
new DistributedCacheEntryOptions in L is tin g 3 fo r y o u r reference.
{
AbsoluteExpirati onRelativeToNow
I m p le m e n t in g D is t r i b u t e d C a c h in g u s in g S Q L S e r v e r
= new TimeSpan(0, 0, seconds: 60)
You can use SQL Server as a cache s to re as w e ll. To g e t
} );
} s ta rte d on th is , create th e database ta b le to sto re cached
data in SQL Server.
return authors;
} ); You can ta k e a dv an ta ge o f th e sq l-cach e by c re a tin g a
com m and to create a database ta b le fo r s to rin g cached
d ata . I f th e sq l-c ac he t o o l is n 't in s ta lle d on y o u r co m ­
p ute r, y o u can in s ta ll i t using th e fo llo w in g com m and a t
th e D eveloper Com m and P rom pt w in d o w :

dotnet tool in s ta ll --global dotnet-sql-cache

Using Distributed msiexec / i "<<Specify the path of the NCache in s ta lle r
Caching in SQL Server in your computer>>"\ncache.ent.net.x64.msi" You can use th e fo llo w in g com m and to create a database
ta b le fo r s to rin g cached data in a SQL Server:
To work with distributed 1. Once th e in s ta lle r is lau nc he d, y o u can observe th re e
caching using SQLServer, d iffe r e n t in s ta lla tio n ty p e s : Cache Server, D e v e lo p e r/ dotnet sql-cache create "Data Sources;
you must have the Microsoft.
QA, and Rem ote C lie nt. I n it ia l Catalog=MyCache;
Extensions.Caching.SqIServer
2. S e le ct CacheServer and c lic k N ext to m ove on . Integrated Security=True;" dbo MyCacheData
NuGet package installed.
3. S p e cify th e License key fo r th e ve rs io n o f NCache
yo u're in s ta llin g in y o u r system . To en able a d is trib u te d SQL Server cache service in an
4. I f y o u d o n 't have a v a lid License key w ith y o u , c lic k ASP.NET Core a p p lic a tio n , w rite th e fo llo w in g piece o f
on th e "G e t In s ta lla tio n Key" b u tto n to g e t an in s ta l­ code in th e Program .es file :
la tio n key.
5. You're p ro m p te d to e n te r y o u r nam e, y o u r o rg a n iz a ­ builder.Services.
tio n 's nam e, and y o u r e m a il address. AddDistributedSqlServerCache(
6. S p e cify th e lo c a tio n in y o u r c o m p u te r w here NCache options =>
sh ou ld be in s ta lle d . C lick on N ext to m ove on to th e {
n e x t ste p. options.Connectionstring =
7. S e le ct th e IP address to b in d th e NCache server. You builder.Configuration.
can s tic k to th e d e fa u lt here. GetConnecti onStri ng
8. S p e cify th e a c c o u n t to run NCache in y o u r system . ("MyDBConnectionString");
S e le ct th e Local System a c c o u n t and c lic k N ext to options.SchemaName = "dbo";
m ove on . options.TableName

66 Distributed Caching: Enhancing Scalability and Performance in ASP.NET 8 Core codem ag.com
 L is tin g 3 : The Program .es file

using Alachi s o ft. NCache.Caching.Di s t r i buted; {

using Microsoft.Extensions.Caching.Distributed; Authorld = 1, FirstName = "Joydip",
LastName = " K a n jila l" , IsActive = true
var builder = WebApplication.CreateBuilder(args); };

/ / Add services to the container. authors = new List<Author>();

authors.Add(user);
builder.S ervi ces.Add NCacheDis t r i butedCache
(configuration => await cache.SetStringAsync
{ (cacheKey, System.Text.Json.
configuration.CacheName = "demoCache"; JsonSerializer.
configuration.EnableLogs = true; S e ria lize (a u th o rs),
configuration.Except!onsEnabled = true; new Di stributedCacheEntryOptions
} ); {
AbsoluteExpirationRelativeToNow
builder.Services.AddH ttpClient(); = new TimeSpan(0, 0, seconds: 60)
var app = b u ild e r.B u ild Q ; } );
}
app.MapGet("/Test", async
(IDistributedCache cache, return authors;
IHttpClientFactory httpClientFactory) => } );
{
const s trin g cacheKey = "Test"; app.RunQ;
List<Author>? authors = n u ll;
var cachedData = class Author
await cache.GetStr1ngAsync(cacheKey); {
public in t Authorld { get; set; }
i f ( ! string.IsNuUOrEmpty(cachedData)) public string? FirstName { get; set; }
{ public string? LastName { get; set; }
authors = System.Text.J son.JsonSerial 1zer. public bool IsActive { get; set; }
Deseri a lize <Lis t <Aut ho r >>(cachedData);
}

else
{
Author user = new AuthorQ

= "MyCacheData"; e x te rn a l resources. M em cached is m a in ly used to reduce

}); th e Load on database servers and im p ro v e th e o v e ra ll e f­
fic ie n c y o f a p p lic a tio n s by s to rin g th e re su lts o f d a ta ­
base queries and o th e r c o m p u ta tio n a lly in te n s iv e re su lts
Now re fe r to th e D istrib uted C ac h in g D e m o ASP.NET Core in m em ory. Data fo r su bs e qu e nt re d u n d a n t queries can
p ro je c t y o u cre ate d e arlie r. Create a new class calle d Prod­ be served fro m th e cache by s to rin g in fo rm a tio n in th e
u c t in a file nam ed P roduct.cs and replace th e d e fa u lt cache, a v o id in g th e database access tim e and c o m p u ta ­
ge nerate d code w ith th e fo llo w in g code: tio n a l overhead. This s ig n ific a n tly im p ro ve s pe rfo rm ance
and s c a la b ility com pared to accessing th e o rig in a l data
public class Product sources fo r every requ est.
{
public Guid Id
{ get; set; }
public s trin g Name Avoiding database access time
{ get; set; }
and computational overhead
public in t Quantity
{ get; set; } significantly improves performance
public decimal Price and scalability.
{ get; set; }
}

Create a new A P I c o n tro lle r nam ed MySQLServerCacheDe- In th is s e c tio n , y o u 'll im p le m e n t a s im p le OrderProcess-

m o C o n tro lle r in a file h a v in g id e n tic a l nam e w ith a .cs in g a p p lic a tio n . To keep th in g s sim p le , th is a p p lic a tio n
e x te n s io n and w rite th e code fo u n d in L is tin g 4 th e re . o n ly d is p la ys one o r m ore o rd er records. The source code
o f th is a p p lic a tio n is co m pris ed o f th e fo llo w in g classes
A R e a l- W o r ld U s e C a s e : I m p le m e n t in g D is t r i b u t e d and in te rfa c e s :
C a c h in g U s in g M e m C a c h e d
M em cached is an ope n-so urce , h ig h -p e rfo rm a n c e , d is tr ib ­ • Order class
u te d , in -m e m o ry ca c h in g system t h a t aim s to im p ro v e th e • IQ rd e rR e p o sito ry in te rfa c e
s c a la b ility and pe rfo rm ance o f w eb a p p lic a tio n s by Less­ • O rderR e pository class
e n in g th e need to re trie v e da ta fro m databases o r o th e r • O rde rsC on trolle r class

codem ag.com Distributed Caching: Enhancing Scalability and Performance in ASP.NET 8 Core 67
 The M ySQ LServerCacheDem oController class
L istin g 4 : kjihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA

[Route( "api/ [c o n tro lle r]")] {

[ApiController] return product;
public class MySQLServerCacheDemoController : }
ControllerBase }
{
private readonly var result = new Product()
IDistributedCache _distributedCache; {
private readonly Id = Guid.NewGuid(),
string key = "MySQLServerCacheDemo"; Name = "HP Envy Laptop",
public MySQLServerCacheDemoController Price = 5000,
(IDistributedCache distributedCache) Quantity = 1
{ };
_distributedCache = di stributedCache;
} byte[] serializedObject
= JsonSerializer.
[HttpGet("GetProduct") ] Seri al i zeTolltf 8Bytes (result);
public async Task<Product> GetProductQ var options =
{ new Di stributedCacheEntryOptions
byte[] cachedObject = {
await _distributedCache. AbsoluteExpirationRelativeToNow
GetAsync(key); = new T1meSpan(0, 0, seconds: 60)
};
i f (cachedObject != null)
{ await _distributedCache.SetAsync
var cachedObjectAsJson (key, serializedObject, options);
= System.Text.Encoding.UTF8. return result;
GetStri ng(cachedObject);
var product = JsonSerializer.
Deserialize<Product>
(cachedObjectAsJ son);
i f (product != null)

C r e a te t h e M o d e l C la s e s s { get; set; }
In a ty p ic a l o rd er m an age m e nt system , you'LL have several public in t Customer_Id
e n tity o r m o d el classes. However, fo r th e sake o f s im p lic ­ { get; set; }
ity , I ' l l co n sid e r and use o n ly one e n tity o r m od el class public DateTime Order_Date
here. Create a new class nam ed Order in a file h a vin g th e { get; set; }
same name w ith a .cs e xte n s io n and w rite th e fo llo w in g public decimal Amount
code in th e re : { get; set; }
}
public class Order
{
public in t Order_Id I n s t a ll N u G e t P a c k a g e (s )
The n e x t ste p is to in s ta ll th e necessary NuGet Package(s).
To in s t a ll th e requ ired package(s) in to y o u r p ro je c t, r ig h t­
L istin g 5: The M em C acheProvider class c lic k on th e s o lu tio n and th e n s e le c t Manage NuGet Pack­

public class MemCacheProvider : IMemCacheProvider ages fo r S o lu tio n .... Now search fo r th e package nam ed

{ Enyim M em cachedCore in th e search box and in s t a ll i t .

private readonly
IMemcachedCli ent memcachedClient; A lte rn a tiv e ly , y o u can ty p e th e com m and show n b e lo w a t
public MemCacheProvider th e NuGet Package M anager Com m and P ro m pt:
(IMemcachedClient memcachedClient)
{
PM> Install-Package EnyimMemcachedCore
thi s.memcachedClient = memcachedClient;
}
public T GetCachedData<T>(string key) You can also in s t a ll th is package by e x e c u tin g th e fo llo w ­
{ in g com m ands a t th e W indow s S he ll:
return memcachedClient.Get<T>(key);
} dotnet add package EnyimMemcachedCore
public void SetCachedData<T>
(string key, T value,
C r e a te t h e M e m C a c h e P r o v id e r C la s s
in t duration)
Now y o u need a class t h a t acts as a w ra p pe r on IM em -
{
memcachedClient.Set cachedCLient so as to e n ca ps u la te a ll ca lls to sto re and re­
(key, value, duration); trie v e da ta to and fro m th e cache. This n o t o n ly s im p lifie s
} access to th e cache, i t also ensures t h a t you d o n 't need to
} w rite code to m anage cache in y o u r c o n tro lle r o r re p o s i­
to ry classes. Now, create a class nam ed M em CacheProvider
and w rite th e code fro m Listing 3 in th e re .

68 Distributed Caching: Enhancing Scalability and Performance in ASP.NET 8 Core codem ag.com
 Listing 6: The O rderR e pository Class

public class OrderRepository: lOrderRepository { };

private public OrderRepository
const string key = "MemCacheDemo"; (IMemCacheProvider memCacheProvider) {
private readonly jnemCacheProvider = memCacheProvider;
IMemCacheProvider jnemCacheProvider; }
private readonly List <Order> orders public async
= new List <Order> { Task <List <Order>> GetOrdersO {
new Order { var result =
Order_Id = 1, jnemCacheProvider.GetCachedData
Customer_Id = 2, <List<Order» (key);
Amount = 125000.00 m,
Order_Date = DateTime.Now i f (result != null) {
}, return result;
new Order {
Order_Id = 2,
Customer_Id = 1, jnemCacheProvider.SetCachedData
Amount = 200000.00 m, <List<Order>> (key, orders, 600);
Order_Date = DateTime.Now return await Task.FromResult(orders);
}, } CBA
new Order {
Order_Id = 3,
Customer_Id = 3,
Amount = 750000.00 m,
Order_Date = DateTime.Now
}

The M em C acheProvider class im p le m e n ts th e IM e m ­ re tu rn e d from th e re ; o th e rw is e , th e d ata is re trie v e d from Namespaces Required

C acheProvider in te rfa c e and w raps access to th e cache th e database. Once th e data is re trie v e d from th e d a ta ­
for NCache
using an in s ta n c e o f ty p e IM em ca ch ed C lie nt. Here is th e base, th e cache is p o p u la te d w ith th is d ata . Lastly, th is
To w o r k w ith N C ache in
source code o f th e IM em C acheP rovider in te rfa c e : data is re tu rn e d :
ASP.NET Core, in c lu d e th e
fo llo w in g na m e s p a c e s in y o u r
public interface IMemCacheProvider public async Task<List<Order>> GetOrdersO p ro g ra m :
{ {
T GetCachedData<T>(string key); var result = jnemCacheProvider. A la c h is o ft.N C a c h e .C a c h in g .
void SetCachedData<T> GetCachedData<List<Order>>(key); D is trib u te d
(string key, T value,
in t duration); if(re s u lt != null) M ic ro s o ft.E x te n s io n s .C a c h in g .
D is trib u te d
} {
return result;
C r e a te t h e O r d e r R e p o s it o r y C la s s }
The O rd erR ep osito ry class uses an in s ta n c e o f ty p e IM e m ­
C acheProvider to in te r a c t w ith th e cache. This in s ta n c e is jnemCacheProvider.SetCachedData
in je c te d in th e c o n s tru c to r o f th e O rd erR ep osito ry class. <List<Order>>(key, orders, 600);
Now, create a new class nam ed O rd erR ep osito ry in a file return await
h a vin g th e same nam e w ith a .cs e x te n s io n . N ext, w rite Task.FromResult(orders);
th e fo llo w in g code in th e re : }

public class OrderRepository : lOrderRepository

{ The c o m p le te source code o f th e O rderR e pository class is
give n in L istin g 6.
}
R e g is te r a n d C o n f ig u r e t h e M e m C a c h e d In s ta n c e
The O rd erR ep osito ry class im p le m e n ts th e m e th od s o f th e The fo llo w in g code s n ip p e t illu s tra te s how an in s ta n c e o f
lO rd e rR e p o s ito ry in te rfa c e . For th e sake o f s im p lic ity , I ty p e lO rd e rR e p o s ito ry is added as a scoped service to th e
have o n ly one m eth o d ca lle d GetOrders. Here is how th e IS e rv ic e C o lle c tio n .
lO rd e rR e p o s ito ry in te rfa c e sh o u ld Look:
builder.Servi ces.AddScoped
public interface lOrderRepository dOrderReposi to ry ,
{ OrderRepository>();
public Task<List<Order>> GetOrdersO;
} N ext, re g is te r and c o n fig u re MemCached using th e fo llo w ­
in g piece o f code in th e Program .es file :
The O rd erR ep osito ry class im p le m e n ts th e GetOrders
m eth o d o f th e lO rd e rR e p o s ito ry in te rfa c e . In th is m eth o d, builder.Services.
y o u f ir s t check w h e th e r th e d ata requested is a v a ila b le AddEnyimMemcached(me =>
in th e cache. I f th e da ta is a v a ila b le in th e cache, it's {

codem ag.com Distributed Caching: Enhancing Scalability and Performance in ASP.NET 8 Core 69
 L is tin g 7 : The C om plete Source o f Program .es file

using Di stributedCachingDemo; {
using Enyim. Caching. Configuration; Address = "localhost",
Port = 11211
var builder = WebAppUcation .CreateBui Ider (a rg s); } );
} );
/ / Add services to the container.
var app = b u ild e r.B u ild ();
b u iId e r. Services.AddControllers();
builder.Services. / / Configure the HTTP request pipeline.
AddScopeddOrderRepository, OrderRepository>();
buiIder.Services. app. UseAuthorization();
AddSi ngletondMemCacheProvi der> (x =>
A ctiva to rU tilitie s.C re a te ln sta n ce app.MapControllers();
<MemCacheProvider>(x));
b u iId e r. Servi ces.AddEnyi mMemcached(me => app.Run();
{
me.Servers.Add(new Server

L is tin g 8 : The OrderControLLer Class In th e f ir s t c a ll to th is e n d p o in t, th e d ata is fe tc h e d from

th e database and i t s fe tc h e d fro m th e cache fo r a ll sub ­
[Route(" api/ [cont r o ll e r ] ")]
[Api Cont ro lle r] se q u e n t calls to th is e n d p o in t. The cache is in v a lid a te d

public class OrdersController : ControllerBase a fte r th e s p e c ifie d tim e s p a n elapses.

{
private ZOrderRepository _orderRepository;
public OrdersController
( IQrderRepository orderRepository) There are a lot of details,
{ but in the end, distributed
_orderRepository = orderRepository;
} caching is fairly simple.
[HttpGet("GetOrders")]
public async Task<List<Order>> GetOrdersO
{
return await _orderRepository.GetOrdersO; Conclusion
} Caching is a proven and e s ta b lis h e d te c h n iq u e used to
} im p ro v e an a p p lic a tio n 's perform a nce , s c a la b ility , and re­
sponsiveness by s to rin g fre q u e n tly accessed d ata in tr a n ­
s ie n t sto ra ge , such as th e m em ory o r o th e r p e rm a n en t
me.Servers.Add(new Server sto ra ge lik e a f ile o r a database. In d is trib u te d c a c h in g ,
{ th e cached da ta is spread across m u ltip le nodes, o fte n in
Address = "localhost", d iffe r e n t n etw o rk s. As cached d ata is re a d ily accessible,
Port = 11211 i t ensures enhance d r e lia b ility d u rin g un ex pe cted peaks
} ); o r fa ilu re s .
} );
J o y d ip K a n jila l
CODE
The c o m p le te source code o f th e Program .es f ile is given
in L is tin g 7 fo r reference.

The OrderController Class

F inally, create a new A P I c o n tro lle r class nam ed Order-
C o n tro lle r w ith th e fo llo w in g code in th e re , as show n in
L is tin g 8 .

The O rde rsC on trolle r class c o n ta in s tw o a c tio n m eth ods,

nam ely, th e GetOrders m eth o d t h a t re tu rn s a List o f Order
in sta n ce s. T his a c tio n m eth o d calls th e GetOrders m eth o d
o f th e O rd erR ep osito ry class re sp e ctiv e ly . N ote how an
in s ta n c e o f ty p e lO rd e rR e p o s ito ry is in je c te d in th e O rder­
s C o n tro lle r class using c o n s tru c to r in je c tio n .

When y o u run th e a p p lic a tio n and brow se th e /g e to rd e rs

e n d p o in t, th e ord er da ta is d is p la ye d in th e w eb brow ser.

70 D is trib u te d C aching: E nh an cing S ca lab ility an d P erfo rm an ce in ASP.NET 8 Core codem ag.com
 O N LIN E Q U IC K ID 2403091 jihgfedcb

C# for High-Performance Systems

My dayjob is writing a database engine named RavenDB. My preferred language to code in is C#.Those two facts tend to cause
glitches in some people's minds. How can you write a database engine, which is all about low-level system programming, in a
high-level language such as C#? How much performance are you leaving on the table by not choosing a proper system-level

language, such as C++ or Rust? I t tu rn s o u t th a t the re is n 't I w a n te d in an e x tre m e ly s u c c in c t and expressive m anner.
a c o n flic t between th e tw o . You can w rite high-perform ance The fa c t t h a t I can do t h a t so s im p ly is one o f th e g re a t
code in C# th a t m atches or beats th e perform ance o f na tiv e s tre n g th s o f C#, in m y o p in io n . B u t g o o d -lo o k in g code
code. The best p a rt is th a t you can achieve th a t w h ile re ta in ­ is ju s t th e f ir s t stage , Let's see how w e ll i t perfo rm s a t
in g many o f th e advantages th a t C# offers in th e fir s t place. ru n tim e .

C# has a lo t o f expressiveness and h ig h -le v e l co nc ep ts , When e v a lu a tin g th e pe rfo rm ance o f a piece o f code, it's
b u t fro m th e g e t-g o , i t has som e c r itic a l fe a tu re s fo r lo w - im p o rta n t n o t to lo o k a t ju s t a s in g le param eter. It's easy
le v e l system s. Value ty p e s , p o in te rs , and unsafe code to tra d e o f f e x e c u tio n tim e fo r m em ory usage, fo r exam ­
have a ll been p a rt o f C# fro m its i n it ia l c re a tio n . R ecent ple. I'm ru n n in g a ll th e code in th is a rtic le using .NET 8 .0 Oren Eini
a d d itio n s to th e .NET p la tfo rm , such as spans, v e c to r in ­ in release m ode. Here are th e e x e c u tio n re su lts: ayende@ ayende.com
s tru c tio n s , and hardw are in trin s ic s make i t m uch s im p le r
to w rite h ig h -p e rfo rm a n c e and Low-Level code. • Peak m em ory: 18 GB Oren Eini started his
professional career as a
• T o ta l a llo c a tio n s : 163 GB
developer in 1999 w ith a
I w a n t to be able to ta lk in co nc re te te rm s, so fo r th e re st • E xecution tim e : 8 m in u te s , 19 seconds
strong focus on th e M icro­
o f th is a rtic le , I'm g o in g to foc us on a re la tiv e ly s tr a ig h t­
s o ft and .NET ecosystem.
fo rw a rd ta s k . The code accepts a CSV f ile (g z ip p e d ) and W hy am I m e n tio n in g m em ory a n d a llo c a tio n s in m y m e t­
He has been a M icrosoft
co m p utes a gg re g ate d s ta tis tic s based on th e d a ta . You rics? The va lu e o f " t o t a l a llo c a tio n s " refers to th e a m o u n t
MVP since 2007. Oren's
can v ie w a sam ple o f th e da ta in th e f ile in T a b le 1 . o f m em ory th e system has a llo c a te d o ve r its Lifetim e,
main focus is on archi­
w h ile "p e a k m em ory" represents th e m axim um a m o u n t o f
te ctu re and best prac­
I ge ne ra ted a Large file to te s t how w e ll th is code per­ m em ory used a t any g iv e n p o in t in tim e . When ta lk in g tice s th a t prom ote q u a lity
form s (th e s c rip t to ge nerate th e d ata is a v a ila b le o n ­ a b o u t .NET program s, re du cing a llo c a tio n s te n d s to have softw are and ze ro -frictio n
lin e a t: h ttp s ://g is t.g ith u b .c o m /a y e n d e /e 8 4 d a c 7 6 e 5 d - an o u ts iz e d im p a c t on pe rfo rm ance, e s p e c ia lly i f y o u're developm ent.
c 0 2 3 f6 a 8 0 3 6 7 f6 c 0 1 a c l3 ), and o b ta in e d a f ile w ith 250 ru n n in g w ith m u ltip le th re a d s a t once. T hat's because th e
m illio n records w ith 9 .7 8 m illio n u n iq u e users. The co m ­ m ore a llo c a tio n s y o u do, th e m ore w o rk th e garbage c o l­ Since 2008, he has been
pressed file size is a b o u t 2 .4 GB. I in te n tio n a lly chose a le c to r has to han dle. W ith m u ltip le th re a d s a ll a llo c a tin g b u ild in g th e RavenDB
large size, w h ic h sh o u ld c le a rly d em o n stra te th e d iffe r ­ m em ory, it's easy to reach a p o in t w here y o u 're m o s tly database, a n o n -re la tio na l
ence in pe rfo rm ance betw e en approaches. b u rn in g CPU cycles on garbage c o lle c tio n . database engine w ritte n
in C#. An in te rn a tio n a lly
The ta s k a t hand is to c o m p u te th e t o t a l q u a n tity and The com pressed f ile size is 2 .4 GB, b u t uncom pressed, th e known presenter, Oren

data is a b o u t 3 .7 GB. Given th e a m o u n t o f tim e t h a t th is has spoken a t conferences

p rice fo r each user in th e file . This is a s im p le ta s k and
such as DevWeek, QCon,
th e code to w rite i t is a lm o s t p a in fu lly o bv io us . The in p u t code to o k to ru n , i t was processing und er 8 MB a second.
Oredev, NDC, and YOW!.
is a GZipped CSV file , so th e f ir s t a c tio n is to decom press T hat's re a lly bad. The reason fo r t h a t is t h a t th e code
th e data and ite ra te th ro u g h th e file one lin e a t a tim e , in L is tin g 2 is e x e c u tin g its o p e ra tio n s in a s e q u e n tia l
as show n in L is tin g 1 . m anner. F irst, i t runs th ro u g h a ll th e records in th e file ,
g ro u p in g th e m by th e user ID , and o n ly th e n does i t run
For reference, i f I were to exe cute G z ip R e a d lA U L in e s - th e a g g re g a tio n fo r each user. T h a t m eans t h a t i t has to
A s y n c (in p u t).C o u n tA s y n c () alon e, i t w o u ld ite ra te o ve r re ta in th e e n tire d a ta s e t in m em ory u n t il i t co m p le te s
th e e n tire file and c o u n t th e n um be r o f line s in i t . E xecut­ th e process.
in g th is code on m y m a ch in e y ie ld s th e fo llo w in g re su lts :
In s te a d o f using LINQ, y o u can choose to w rite th e ag ­
• T o ta l a llo c a tio n s : 53 GB g re g a tio n code y o u rs e lf. You can see th e code fo r th is in
• E xecution tim e : 47 seconds L is tin g 3 w here y o u 're b a s ic a lly d o in g th e a g g re g a tio n
in lin e . For each record, y o u lo o k up th e re le v a n t user s ta ­
T his is w ith o u t d o in g any a c tu a l w o rk , m in d . It's ju s t tis tic s and up d ate th e m . You o n ly need to keep tra c k o f
re a din g th ro u g h th e d a ta s e t. th e users' s ta tis tic s , n o t th e e n tire d ata set.

The n e x t ta s k is to a c tu a lly do s o m e th in g w ith th e data

and ge nerate th e ag gre g ate d re su lts . I'Ll s ta rt fro m h ig h - U serID It e m ID Q u a n tity Price D ate
le v e l code and w rite th e a g g re g a tio n using LINQ. Using 22271148092 5203189368466850000 91 7.03 2 4 /1 2 /2 0 2 3 11:02:20
th e S y s te m .L in q .A s y n c package, y o u can a p p ly a LINQ
22271148092 1618666246057250000 2 11.49 2 4 /1 2 /2 0 2 3 11:02:20
expression to th e re su lts o f G zip R e a d lA L L L in e s A s y n c (),
w h ic h w ill do a ll th e c o m p u ta tio n in a very cle a r fa s h io n . 23392313395 7334974373793960000 1 40.69 2 4 /1 2 /2 0 2 3 11:02:20
You can see how th is Looks in L is tin g 2 . 23145412223 2702876167686390000 2 15.56 2 4 /1 2 /2 0 2 3 11:02:20

23145412223 5203189368466850000 2 7.03 2 4 /1 2 /2 0 2 3 11:02:20

Before I discuss pe rfo rm ance, I'd lik e y o u to ta ke a Look a t
L is tin g 2 and see how I was able to w rite p re cis e ly w h a t T a b le 1 : Sam ple da ta fro m th e orders CSV file .

codem ag.com C# fo r H ig h -P e rfo rm a n c e Systems

 What's the result of executing this code, then? It's going saved almost four-fifths (!) of the runtime and are using
to use Less memory, but how much faster is it going to be? about 5% (!) of the memory. That is an amazing differ­
ence by all accounts.
• Peak memory: 917 MB
• Total allocations: 126,999 MB (127 GB) It's also interesting to realize that the total amount of
• Execution time: 1 minute, 51 seconds allocation isn't that drastically different. It's 163 GB vs.
127 GB, even though all the other metrics are far bet­
The amount of code that you wrote isn't significantly dif­ ter. Why is that? Most of the allocations here are fairly
ferent between Listing 2 and Listing 3. Both are pretty unavoidable because you read the lines from the file one
clear and obvious in how they do things. However, you line at a time and generate a lot of strings. Additional
strings are also allocated during the parsing of each line
because you use Split(',').
Listing 1: Decompressing and yielding the lines from the file.kjihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA
async IAsyncEnumerable<string> The code in Listing 3 has far better performance than the
GzipReadlAHLinesAsync(Stream input) { in itia l version, but it can still improve. A key issue is the
using var gzipStream = new GZipStream(input, manner in which you process the data from the file. You're
Compressi onMode. Decompress); creating a lot of strings in the process, which is a big
using var reader = new StreamReader(gzipStream);
waste; you can do better. Instead of allocating strings for
while (true) {
each line of the file, (remember, there are 250,000,000 of
string? lin e ; = await reader.ReadLineAsync(); them), you can use a lower-level API.
i f (lin e == n u ll)
break; System.IO.Pipelines is one such low-level API that's
yie ld return lin e ;
meant to provide an efficient way to read (and write)
}
data in .NET. In particular, it uses a single reusable buf­
}
fer to do the work and allows you to access it directly
instead of performing allocations. The API is somewhat
Listing 2: Computing total and quantity per user using Linq less convenient to use, but the performance difference
more than makes up for this. Take a look at the code in
ValueTask<Dictio n a ryd o n g, UserSales>> Listing 4, showing the skeleton of using Pipelines for
Linq(Stream input) {
processing the file.
return (from I in GzipReadlAHLinesAsync( i nput)
. S kip (l) / / skip header lin e
le t fids = l . S p l i t C , ') Unlike previous code listings, the code in Listing 4 isn't
le t item = new { actually doing much; most of the work is handled by the
Userid = long. Parse (fids [0 ]), ProcessLines() method, which w ill be covered shortly.
Qty = i n t . Parse (fids [2 ]),
What's interesting here is that all the I/O is handled in
Price = decimal. Parse (fids [3])
the parent method. You ask the pipe to read a buffer
}
group item by item.Userid in to g from the stream and then process it. A key aspect is that
select new { the buffer isn't limited to a single line or is guaranteed
Userid = g.Key, to be on a line boundary. Instead, you get some buffer
Quantity = g.SumAsync(x => x.Q uantity),
from the file (64KB in this case) and process everything
Total = g.SumAsync(x => x.Price)
} ) .ToDictionaryAsync(x => x.Userid, in it.
x => new UserSales {
Quantity = x.Q uantity.Result, The really nice thing about the System.IO.Pipeline ap­
Total = x.Total.R esult proach is that it encourages a highly efficient manner for
} ); processing incoming data. Let's take a look at Listing 5
}
where I actually do that and then I'll discuss exactly what
is going on here.
Listing 3: Manually aggregating over the records
The code in Listing 5 isn't really doing much, but it's one
async Task<Dictionary<long, UserSales» of those cases where I removed everything non-essential
StreamReaderAndDictionary(Stream input) {
to expose the core beauty. The PipeReader is going to
var sales = new Dictionary<long, UserSales>(); read from the stream into a buffer, which is held by the
await foreach(var lin e in ReadResult. You use a SequenceReader<byte> to search
GzipReadlAHLinesAsync( i npu t). Skip(1)) { the current line in the buffer and return a reference (to
the same original buffer) to the range of the line. If you
var fields = lin e . S p lit( ' , ') ;
can't find a line break, that means that you need to read
var uid = lo ng.Parse(fields[0 ]);
in t quantity = i n t. Parse (fields [2 ]);
more from the stream, so you inform the pipe reader how
decimal price = decimal. Parse(fields[3 ]); far you read and until what point you've examined the
buffer. The next call to ReadAsync() (shown in Listing
i f ( ! sales.TryGetValue(uid, out var sta ts)) 4), w ill then know what can be discarded and what more
sales[uid] = stats = new UserSalesQ; needs to be read from the stream.
stats.T otal += price * quantity:
stats.Q uantity += quantity; There isn't a lot of code here, but from an architectural
} perspective, there's a lot going on. The end result is that
return sales; you can use a single reusable buffer to process the data.
} The ProcessLinesQ method also contains a loop. It's go­

72 C# for High-Performance Systems codemag.com

ing to be running over the entire buffer, which is very Listing 4: Reading from the file using PipeReader
Likely to contain many Lines. Instead of operating on a
async ValueTask<Dictionarydong, UserSalesStruct>>
per-line basis, it's now operating over a far bigger batch
PipelineAsync(Stream input) {
of items. That also has implications with regard to per­ using var gzipStream = new GZipStream(input,
formance, because the hot Loop needs to cover a lot less Compressi onMode. Decompress);
code to do its job. The actual processing of the lines is var pipeReader = PipeReader.Create(gzipStream,
shown in Listing 6, which also contains some interesting new StreamPipeReaderOptions(
bufferSize: 64 * 1024)
code patterns to talk about.
);

I'm afraid that the code is pretty dense, but the idea is var header = false;
pretty simple. You use another SequenceReader<byte> var salesData = new D ic tionarydong, UserSalesStruct>();
to find the terminators in the line buffer you're given while (true) {
var valueTask = pipeReader.ReadAsyncO;
here, you then use Utf8Parser to parse them without
var read = valueTask.IsCompleted ? valueTask.Result :
needing to allocate any strings. The code is complicated await valueTask.AsTask();
because it's a single expression and because I wanted to ProcessLines(pipeReader, read, salesData, ref header);
produce the correct error in the case of invalid input. Note i f (read.IsCompleted)
that I'm using an unconditional bitwise and (&) versus break;
the more common condition and (&&). This ensures that }
the entire expression runs and avoids branch instructions.
return salesData;
This is the hotspot in the code, so anything that reduces }
costs is advisable, and branches can be costly (even if
well predicted, as in this case).

I'm also not using the Dictionary as you'd normally ex­ Listing 5: Processing lines in the buffer
pect. I'm calling CollectionsMarshal to get or add the void ProcessLines(PipeReader pipeReader,
value. I'm not sure if you noticed, but Listings 4, 5, ReadResult readResult,
D ic tionarydong, UserSalesStruct> salesData,
and 6 all use UserSalesStruct as the value, instead of
ref bool header) {
UserSales (a class). That change allows you to avoid al­
locations even further and take advantage of the Get- var sr = new SequenceReader<byte>(readResult.B u ffe r);
ValueRefOrAddDefault() behavior. while (true) {
ReadOnlySequence<byte> lin e ;
A very typical access pattern when using dictionaries is i f (sr.TryReadTo(out lin e , (b y te )’ \ n ’ ) == false) {
pipeReader.AdvanceTo(consumed: s r . P o sition,
to look up a value by key and create it if it isn't in the
examined: readResult.Buffer.End);
dictionary. This requires you to do two dictionary look­ break;
ups if you need to create the value. The GetValueRef }
OrAddDefault() means that you only need to do one. i f (header == false) {
Because a struct in C# is guaranteed to be zero in itia l­ header = true;
continue;
ized, you don't care if the value exists or not and you can
}
immediately add the current record values to the user's ProcessSingleLine(salesData, lin e ) ;
sales. }
}
Finally, you might have noticed that the meth­
od in Listing 6 has an interesting attribute:
[ MethodImpl(MethodImplOptions.AggressiveInlin- Listing 6: Processing a single Line
ing)]. This is an instruction to the Just In Time compiler
[Methodlmpl(MethodlmplOptions.A ggressiveinlining)]
to force inlining of this method to its caller. This allows
void ProcessSingleLine(
you to benefit from clear code and separation of responsi­ D ictionarydong, UserSalesStruct> salesData,
bilities while getting dense (and efficient) machine code. ReadOnlySequence<byte> lin e )
{
The Pipeline version is spread over three code listings, var I r = new SequenceReader<byte>(line);
ReadOnlySpan<byte> span;
and I had to reach quite far out of the beaten path. What
var readAll = lr.TryReadTo(out span, ( b y te ) ', ')
did you get, then, from this journey? Take a look at these & Utf8Parser.TryParse(span, out long userid, out _)
results: & Ir.TryAdvanceTo((b y te )’ , ')
& I r .TryReadTo(out span, (b y te )' , ')
• Peak memory: 846 MB & Utf8Parser.TryParse(span, out in t quantity, out _)
& lr.TryReadTo(out span, ( b y te ) ',')
• Total allocations: 1069 MB
& Utf8Parser.TryParse(span, out decimal price, out _);
• Execution time: 50.5 seconds
i f (readAll == false)
I triple checked those results because they are really throw new InvalidDataException(
good. You can see the fu ll results in Table 2, but the "Couldn't parse expected fields on: " +
Pipelines method is almost 50% faster than the Stream- Encoding.UTF8.GetString(line));

Reader + Dictionary approach you used in Listing 2. It's

ref var current = ref CollectionsMarshal
also Literally allocating less than 1% of the memory. .GetValueRefOrAddDefault(salesData, userid, out J ;
current.Total += price;
The task of processing a file and aggregating the results current.Q uantity += quantity;
is a simple one. The sort of thing that's commonly given }

codemag.com C# for High-Performance Systems 73

 CODE COMPILERSjihgfedcbaZYXWVUTSRQPONMLKJIHGFED

Scenario Time (sec) Allocations Peak memory

(MB) (MB)
L in q 49 9 1 6 6 ,6 6 0 1 8 ,2 8 6
Y E A R S
M a r/A p r 2 02 4
S tream Reader + D ic tio n a ry 111 1 1 7 ,2 7 2 917 V o lu m e 2 5 Is s u e 2

P ip e lin e s 5 0 .5 1 ,0 6 9 84 6
Group Publisher
M a rk u s E gger
S um m ary o f pe rfo rm ance and m em ory c o n s u m p tio n across a ll scenarios
T ab le 2 : kjihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA
Editor-in-Chief
Rod P ad doc k

Managing Editor
as course ass ig nm en ts to s tu d e n ts . I t s also a p re tty good
E lle n W h itn e y
scen ario to te s t o u r c a p a b ilitie s because i t s so lim ite d
Content Editor
in scope. The p o in t o f th is a rtic le is n 't to p o in t y o u t o ­
M e la n ie S p ille r
w ard th e S y s te m .IO .P ip e lin e s A P I (a lth o u g h I ' l l a d m it
Writers in This Issue
th a t it s a fa v o rite o f m in e ), i t s to d e m o n s tra te how P h ilip p B a u e r M a rk u s E gger
y o u can e n tire ly change th e m an ner in w h ic h y o u p er­ O ren E in i J o y d ip K a n jila l

fo rm a ta s k fro m th e e x tre m e ly h ig h -le v e l LINQ expres­ W e i-M e n g Lee S a h il M a lik

S h a w n W ild e rm u th M ik e Y ea g er
sion a ll th e w ay dow n to reusable b u ffe r and a lm o s t no
Technical Reviewers
a llo c a tio n s .
M a rk u s E gger
Rod P ad doc k
C# and .NET a llo w y o u to create s o lu tio n s fo r a very
Production
w id e range o f scenarios. From b u ild in g a business ap ­ F rie d l R a ffe in e r G ra fik S tu d io
p lic a tio n w ith a foc us on corre ctne ss and tim e to m ar­ w w w .f r ig r a f . it

ke t, a ll th e w ay dow n to database eng in es such as Graphic Layout

RavenDB. In RavenDB, y o u 'll fin d t h a t th e re is no such F rie d l R a ffe in e r G ra fik S tu d io in c o lla b o r a tio n
w it h o n s ig h t ( w w w .o n s ig h td e s ig n .in fo )
th in g as " fa s t e n o u g h ." You can spend lite r a lly m o nths
Printing
o p tim iz in g a s in g le piece o f code to be ju s t a L ittle b it
Fry C o m m u n ic a tio n s , In c .
fa ster. 8 0 0 W e st C hurch Rd.
M e c h a n ic s b u rg , PA 1 7 0 5 5

You're m a kin g a Lot o f use o f s e e m in g ly e s o te ric fe a tu re s Advertising Sales

such as hardw are in trin s ic s , v e c to r o p e ra tio n s , and zero T a m m y F e rg uso n
8 3 2 -7 1 7 - 4 4 4 5 e x t. 26
a llo c a tio n AP Is. The nice th in g a b o u t t h a t is t h a t y o u can
ta m m y @ c o d e -m a g a z in e .c o m
m ee t y o u r pe rfo rm ance goals w h ile s t i l l w r itin g in C#.
Circulation & Distribution
G e n e ra l C ir c u la tio n : EPS S o ftw a re C orp.
T h a t is n 't ju s t a pe rs on a l preferen ce o f w a n tin g to sta y N e w s s ta n d : In g ra m P e rio d ic a ls , In c .
in y o u r c o m fo rt zone. I s ta rte d w r itin g C# code because I n t e r n a t io n a l B o n d e d C o u rie rs (IB C )
M e d ia S o lu tio n s
o f th e ex perie nce o f g e ttin g an e rro r w ith a p ro p e r e rro r
S o u rc e I n t e r li n k I n t e r n a t io n a l
message and a d e ta ile d sta c k tra ce , ra th e r th a n som e hex
Subscriptions
code and "y o u fig u re i t o u t. " T h a t exp erie nce is s t i l l very
Circulation M an ag er
m uch tru e to d a y : U sing C# as th e p rim a ry language fo r
C o lle e n Cade
RavenDB m eans t h a t y o u can use a ll o f th e in fra s tru c tu re 8 3 2 -7 1 7 - 4 4 4 5 e x t. 2 8
around .NET fo r p ro filin g , d e b u g g in g , and m o n ito rin g . c c a d e @ c o d e m a g .c o m

There's also th e q u a lity o f th e to o lin g aro und .NET t h a t

US s u b s c r ip tio n s are $ 2 9 .9 9 USD f o r o n e y e ar.
m a tte rs a lo t. S u b s c rip tio n s o u ts id e th e US are $ 5 0 .9 9 USD.
P ay m e n ts s h o u ld be m a de in US d o lla r s d ra w n
o n a US b a n k . A m e ric a n Express, M a sterC a rd ,
Beyond e x p lo rin g a s m a ll w ay in w h ic h y o u can im p ro v e
V isa a n d D is c o v e r c r e d it c a rd s a c c e p te d .
pe rfo rm an ce by an ord er o f m a g n itu d e , I w a n t to em ­ B ack is s u e s are a v a ila b le . F or s u b s c r ip tio n
phasize t h a t perfo rm an ce is a jo u rn e y . You need to a lig n in f o r m a tio n , e m a il s u b s c r ip tio n s @ c o d e -m a g a z in e .c o m

y o u r o v e ra ll a rc h ite c tu re and th e a lg o rith m s y o u use w ith o r c o n ta c t c u s to m e r s e rv ic e a t 8 3 2 - 7 1 7 - 4 4 4 5 e x t. 9 .

th e a c tu a l q u a lity o f im p le m e n ta tio n and m ic ro -o p tim iz a ­ Subscribe online at

tio n s . U sing RavenDB as an exam ple a g a in , it's com m on , w w w .c o d e -m a g a z in e .c o m

in c e rta in h o ts p o ts , to m o d ify (C#) code in ord er to g e t

CODE D e v e lo p e r M a g a z in e
th e r ig h t m a ch in e code fo r y o u r needs. It's q u ite am azing EPS S o ftw a re C o rp o ra tio n / P u b lis h in g D iv is io n
t h a t y o u can do t h a t w hen y o u need to . D oub ly so, as yo u 6 6 0 5 C ypressw ood D rive, S te 4 2 5 , S p rin g , Texas 7 7 3 7 9 USA
Phone: 8 3 2 -7 1 7 - 4 4 4 5
can do t h a t fo r b o th x 6 4 and ARM m achines.

F inally, I ' l l Leave y o u w ith th is to consider. It's o fte n b e st

to go th e o th e r w ay aro un d. Look a t th e b e s t pra ctic es fo r
h ig h -p e rfo rm a n c e im p le m e n ta tio n s and th e n w a lk back­
w ard to an a rc h ite c tu re t h a t w o u ld en able y o u to use
th a t . W ith t h a t done, y o u can th e n s e le c tiv e ly im p le m e n t
th e h o ts p o ts as needed, w ith o u t h a vin g to s tru g g le w ith
p a st a rc h ite c tu ra l decisions.

Oren Eini
CODE

74 C# for High-Perform ance System s

DCBA

C US TO M SOFTW ARE D E V E LO P M E N T
S T A F F IN G T R A IN IN G /M E N T O R IN G S E C U R ITY

M O R E T H A N JU ST
A M A G A Z IN E !
D o e s y o u r d e v e lo p m e n t te a m la c k s k ills o r tim e t o c o m p le te all y o u r b u s in e s s -c ritic a l s o ftw a r e p ro je c ts ?
C O D E C o n s u ltin g has t o p - t ie r d e v e lo p e r s a v a ila b le w ith in - d e p th e x p e rie n c e in .N E T,
w e b d e v e lo p m e n t, d e s k to p d e v e lo p m e n t (W P F ), B la z o r, A z u re , m o b ile a p p s , lo T a n d m o re .

C O N T A C T U S T O D A Y F O R A C O M P L IM E N T A R Y O N E H O U R T E C H C O N S U L T A T IO N .
k
N O S T R IN G S . N O C O M M IT M E N T . J U S T C O D E .
c
o
t
sr
e
tt c o d e m a g .c o m /c o d e
u
h
s
© 832-717-4445 ext. 9 • info@ codem ag.com
w

CODE STAFFINGjihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA

UNLOCK ±
S T A F F IN G *
EXC ELLENC E
T o p -N o tc h IT T a le n t, C o n tr a c t F le x ib ility , H a p p y T e a m s , a n d a
C o m m itm e n t t o C u s to m e r S uccess C o n v e r g e w it h C O D E S ta ffin g

Our IT staffing solutions are engineered to drive your business forward while
saving you time and money. Say goodbye to excessive overhead costs and
lengthy recruitment efforts. With CODE Staffing, you'll benefit from contract
flexibility that caters to both project-based and permanent placements. We
optimize your workforce strategy, ensuring a perfect fit for every role and
helping you achieve continued operational excellence.

Ready to Discuss Your IT Staffing Needs?

Visit our website to find out more about how we are changing
the staffing industry.

W ebsite: codestaffing.com
Y air A lan G riv e r (yag)
C hie f Executive O ffic er
D irect: +1 425 301 1590
Email: yag@ codestaffing.com

76/76