Definition of a Dista ee nce Measure, Euclidean Distances, J i i ince. ss, Jaccard Distance, Cosine Distance, is called a space. A space is necessary to define any distance measure. Let x and y be two: ,, then a distance measure is defined as a function which takes the two points x and y a5 ces the distance between the two points x and y as output. The distance function is denoted luced by the function d is areal number which satisfies the following axioms: = = 7 The distance between any two points can never be negative: J tance: The distance between a point and itself is zero, nce from y to X. P é between x and y is always smaller than or equal to the ity : The direct distance tz. In other words, distance measure is the length of the een x and y via another point between two points x and ¥- x, z) + d(z, y)Coys orld (eetele| Hy) are sq final result= O61 + 4-7 443 7 max ({10 - 6), Ja -7)) max (4,3) rd similarity which measures the cosenes of two ses Jccard Beare) ara the size of the union of the sets x andy axioms on the Jaccard distance : + The size of the intersection of two sets can never be more than the size ratio SIM(x y) will always be a value less than or equal to 1. Ix y, then xu x= xx = x In this case SIMO 9) = WKS fe the Jaccard distance between the same set and itselfis zero, both Union as well as intersection are symmétrick SY = ys and xy nce is also symmetric dx y) = 40 X> 1d distance can also be considered a6 the probly 3 Jaccal yto the same value, an does not map both the sets x and h(x) + hiz)] + Phi) * hy) random minhash_ function.aes eee 0z0c 6Y OcFinding Similar Iter 0z0z 6V OddO A Common Subsequence (LCS) method stance between two strings x and y is the least number of edit operations required to y. Only two edit operations are allowed : are. of a single character, and % | of a single character. ; P jet us take the following two strings : nf x = JKLMN y = JLOMNP ail - the Edit distance between x and y we have to convert string x into string y using the edit of insertion and deletion. re the character sequences in both the strings : w f= ) kK Saban y = J -b yOu@M Ny ® Loot plato, ©2008 © Positions 4 @,@and@are having diferent characters in x and y. So we need to make the ions and deletions at these three positions. KL @ Lo P q ne a @oo ae i @ of string x, we have to delete the character K. The characters following K will be tion e position to the left. ; the status of the string x is : edit operation (deletion)fe} be) 0 [o} 4 © iN co} N ° The edit distance for converting stn as the sequence of insertions sya np x into string y will be the same for converting string leletions can be reversed, lity : The sum of the i f '¢ number of edit operations required for converting string x into String z into string q y can never be less the ‘i c ‘ Sting x direct into the svng y 'an the number of edit operations required for ce measure with an example. © is applicable in the space of vectors. Hamming distance between two vectors is the ints in which they differ from each other. 4s consider the following two vectors : ce= 1.0.0 Ogle m= 1 i 1 0 oO teu ude @ ®@ © @ © © -Positions sance between the above two vectors is 3 because components at positions 2, 3 and 6 are mming distance may be verified as follows : - ‘on the Ha s will differ in at least zero or more component positions, so, the : Any two vector ance can never be negative. two identical vectors, I be the same whether x is compared with y or y is compared the Hamming distance will be zero. + Only in the case of + The Hamming distance wil of differences between x and % plus the number of differences + The number ee: me 3s between x and y. ‘and y can never be less than the number of difference:St It is vast widely used now-a-days. It is likely a subclass of information filtering system. It is used to g recommendations for books, games, news, movies, music, research articles, social tags etc. ‘+ Itis also useful for experts, financial services, life insurance, and social media like Twitter etc. * Collaborative filtering and content-based filtering are the two approach used by recommendation system, * Collaborative filtering uses user's past behaviour and apply some predication about user may like accordingly post data. ‘+ Content based filtering uses user's similar properties of data preferred by user. * By using collaborative filtering and content based filtering a combine approach is developed ie. recommendation system. 9.1.1 The Utility Matrix ‘+ Arecommendation system prefers the preference of a utility matrix. Users and item’s these are entities by recommendation system. * Users have preference to data and these preferences must be observed, + Every data itself is part of utility matrix as it belongs to some item category. Example : A table representing users rating f SBS Gti 8Sealeld to S, with 5 as highest rail represents that user not replied on Seal Al, A2 a3 fOr Androld Ty 2'snd 31,1213 for 105 1. 208 B and C gives rating.for some apps there is less number of responses matrix is to make some predictions for blank spaces, these prediction are useful in Q 5 to i2 app so we have to take in i Account parameters of app i2 like its GUI, memory ity, music/effects if applicable etc. s Fating 5 to A2 app so we have to take similar Parameter in consideration. By judging and all we can put predication what can be further recommended to user A and B. Se though there is no use of full rating anywhere stil it can be judge and predicted sed app user'c’ should be recommended. sovit online good seller/ouyer, trading website uses recomme n, Movie Recommendation, News Articles etc. areW Big Data Analytics (MU) items. 9.2.1 Item Profile *+ An actor of drama or of movie is considered as an actor set, Few favourite actor(s) 7“ . A set of teachers, some students prefer to be guided by few teacher(s) only. ° The year in WiC cor made. Few viewers prefer old songs; some prefer to late only, users sorting of songs based on year. * So many classes are available which provides some data. * Few domains has common feature for example a college and movie it has students, professors actors, directors set respectively, Certain ratio is maintained as many student and few professors in g while many actor works under one or two director guidance. Again every college and movie has datasets as movie released in a year by director and actor and college has passing student every year * Music (song album) and a book has same value feature like songs writer/poet, year of release and Publication year respectively. viewers prefer drama or movielues of count founds threshold becomes a part of feature set. it a document. To know similarity between any two documents, there distance en sets it can be done by either res from Tags database many features are available like Title, ISBN, Edition, Printing, Price, opyright etc, but user (reader) of book mainly concern with tile and price tag mostly. ‘of features of same items by tag items by entering phrase or search value range for at item on feature of tag value like price range for book, available, users can search etc in online shopping. isto create tags and such enough tag awareness at user level Profile he rating for mobile application based on Android >» Tae make a sense to get possible een erical for instance we might take t re. mb: like 1, 2, 3, 4 and nt like 1 star or 5 stars will not 5 stars by some users of it9-5 fe average for rating can be observed without lose gp Wig Data Analytics (MU) - = By keeping 5 options in rating good possibl implicit in numbers. Numerical value based rating provides a si Boolean value and other integer valued or features tell about similarity of items. 9.2.5 User Profiles ingle component of vector representing to items real-valued parameter becomes components of vectors «Vectors are useful to describe items and user's preferences. Users and items relation can be plotted help of utility matrix, Example : Consider similar case like before but util 5 range. Consider, user U gives responses with average based games) got ratings of 3, 4 and 5. Then user profile of ie. rated average of 3-3, 4-3 and 5-3 i.e. value of 1 + Onother hand, user v gives average rating 4. So user v responses to application are 3, 5 and 2. ‘©The user profile for v has in the component for application, the average of 3-4, 5-4 and 2-4, i. valu 9.2.6 Recommending Items to Users based on Content ‘+ Between user's vector and item's vector cosine distance can be computed with help of profile vec users and items both. ‘+ Itis helpful to estimate degree to which user will prefer as an item (i. prediction for recommendatio + If user's and response (like 1 to 5 scale for mobile apps) vectors cosine angle is large positive means angle is close to 0 and hence there is very small consine distance between vectors. + If-user’s and responses vector cosine angle is large negative fraction. It means angle is close to d 1180 which is a maximum possible cosine distance. + Cosine similarity function = for measuring cosine angle between two vector lity matrix has some nonblank entries that are rate of 3 there are three applications ( U, the component for application will ha cose = TG * Invector space model Va = IW, d,W, d, ... WNd]7 Where, W,4: TF# IDF weight of term t’in‘d' document TF : Term Frequency 5 IDF: inverse document frequency mis “eg au 9.2.7 Classification Algorithm Itis used to know user's interest. By applying some fi aed may like. we get some probability whiRecommendation Systems occhio's algorithm '9 as few domains are used move by like human-computer interaction, information ‘retrieval system andA For example, user A and User p 1/5 and Jacard dist te aiscard similari — arity 2/4 and Jaccard gf Rec distance ig jommendation Systems e closer than A and B, same ie. y, ting from 1 to 5 to any 0) i Any app then it is considered = as a0 (zero) Cas longer cosine value implies a smaller angle. 1g and assign NULL to lower values. one value to higher ratin y matrix few apps having ratings like 3, 4 and 5 will consider it as “1' and bee II consider it as unrated keep it NULL ct conclusion 2s Jaccard distance between A A compared to B. . conclusion and it can be verified by applying. cosine distance @ 6 and Bis 3/4 and A and Ci 1Big Data Analytics (MU) 9.4 Pros and Cons in Recommendation System (No knowledge engineering efforts needed. i) Serendipity in results. (ii). Continuous learning for market process. Cons ()__ Rating feedback is required. (i). New items and users faces to cold start. 9.4.2 Content-based Filtering Pros @ No. community requirement. (i) Items can be compared among themselves. Cons (@ _ Need of content description. Gi) New users face cold start. 9.5 Case Study: Product Recommendation Here are some use cases of Recommendation Engines : * According to a recent study, around 75% of Netflix consumers watch contents that are recommende system, The Netflix executives say that their recommendation system saves about $1 billion each * Amazon is also making 35% more revenue using recommendation systems. They credited a 29% incr sales to the use of recommendation systems which is approximately over $135 billion. + Best Buy, another big retailer, reported a 23.7% increase in sales using recommenders. * Linkedin, the professional social network, also uses a recommendation system to recommend u: or newsletters to the users for better user experiences. * Youtube's recommendation engine is powered by Google Brain. Google shows advertisements di videos using recommender systems and eam a revenue of more than $22 blion.Recommendation Systems ican technology gi; SY giant that offers services such ag e-commerce, cloud computing, and founded by Jeff 8 in : by Jeff Bezosin 1994. and ig Considered the largest online marketplace in the d a recommendati it dation system across its e-commerce platform almost two decades ago. g this system, Amazon used to make ler lists, Product recommendations to users based on human g to Amazon, this manual approach wa: de recommendations to users with niche int s found to be inherently biased and did not rests, developed and deployed an automatic algorithmic system that matches a user's fated items with similar items, and then combines these similar items into 2 list of lucts for a user. roach of making recommendations became known as “item-based collaborative filtering” oF ollaborative filtering.” n uses recommendation algorithms to offer different categories of recommendations to users tures on its e-commerce platform. These categories are listed down below : { For You : When any user visits the Amazon webpage andlogs in, they will see a tab on the is tied to their account. Once they click on this they are presented with a wide range of dations across multiple categories. For example, this can include recommendations ks, apparel, electronics or other categories. atte ett Together : In order to increase the average order value, Am: on items that have been purchased together in the past by other aim to convince the customers to purchase additional items ( also13. a4 as as as Big Data Analytics Related To/Based On Items You Vi recer very similar to items that a user ré é hangers on the platform, these recommendations atc. These recommendations are also based on 2 Us setomers Who Bought This tem Also Bought : This Cofeoe : a been purchased collectively by other users in the past. This category of recommendations ve the Frequently Bought Together section and it similarly aims to increase average order value selling and cross-selling as described above. Recommended Items Other Customers Often Buy Again : This category of recommend items that similar users (based on purchase history) often purchase multiple times. New Version of This Item : This category of recommendations is based on the assumption that like to upgrade items, such as mobile phones, that they purchased. As 2 result, ths recommendations informs users when a new edition of an item they purchased is available. Recommended For You Based on a Previous Purchase/Inspired By Your Purchases/Inspi Shopping Trends : These categories of recommendations make product suggestions to a user recent purchase they have made, After a user makes a purchase on Amazon, they are directed details page. On this page, the user will receive further recommendations for items that can be the initial item order. For example, if a user purchases an iPhone on Amazon, they migl recommendations for iPhone covers on the subsequent order details page. These recommend appear on the homepage. This category of recommendations aims to encourage users to make purchase by offering a relevant cross-sell offer. Best-Selling in Different Categories : This category of recommendations features top-selling it the different categories of products on Amazon. It is based on the idea that an item that has purchased by other users is validated as worthwhile. In addition, these recommendations aim to h identify popular products and make purchases from new categories of products that they have purchases from before. This produces a range of up-sell and cross-sell opportunities for the Amazo Popular in Brands You May Like : This category recommends items that are popular and are brands or sellers that a user may be interested in based on previous searches/purchases. Off-site Email Recommendations : Amazon also sends users product recommendations via can opt out of receiving these marketing emails, and can also select certain categories of items (e.g. books, Amazon Echo) for which they would like to receive the marketing emails. These emails vary content and focus, but contain similar categories of recommendations as those available on Amazon, cred ; These categories of recommendations Suggest Prog ntly viewed on the website or app- For example, if a user u would suggest hangers of different shapes, er’s recent browsing history. ry of recommendations suggi What is recommendation system Enlist application of recommendation system and taxonomy for application recommendation system. Explain utility matrix with example, = Explain item profile of content based recommendation, How item profile is represented 7Is one of social network which Ces following Featured on Google. + etc, graph is collection of set of edges (e) and set of vertices (V).If there | des of graph then that node relates with each other, categories by many parameters like ordered pairs of no ‘has direction, weight. Relationship among graph aeMining Social Ne 10-3, Big Data Analytics (MU) 10.2.1 Parameters Used in Graph (Social Netwo © Biplane graph sisted be (i) Degree (i) " [2 _How degree, closeness, | general para set of links. Some Every node is distinct in a network and it i part of graph by for any social network as graph are — (a) Degree : Number of adjacent nodes (considering both out degre ae denoted by din). u'ne (©) Geodesic Distance : Actual distance between two node ny and ny expressed By (©) Density : It gives correctness of a graph, itis useful to count closeness of network. ;: (4) Centrality : It telis about degree centrality ie. nodes appearance in the centre of network c types. Example: Amit Amar Mahesh Rahul ‘Sachin Fig. 10.2.1 Degree of each nodes are as follows : eicren® 26 ah 22 Rahul 4 Sachin [1 Density of undirected graph is 0.6. Geodesic Distances between two nodes is as follows : tn ee = bb = lpia L 2 Mahesh A 2 a Wi 2 witMining Social Network Graph Cy = toy (g-1) SM) = C (ayig-1) P Se " = Gin) i[s=ta=2) Dy jek IU /I 9x = The number of geodesics connecting jk ‘The number that actor iis on. @ each node has some value and as it gets connected with another node its values get ° ‘has some records on his name in single. There are some other records on his name nother player name in doubles. 7 e different values depending on its connection with neighbouring node. ta are available having two or more common networks. Network os “represents an Email account it isa single node. Every node of an e-mail is in link with at least (ie. sender mail ID and receiver mail ID). nail are send from one side and sometime e-mail are send from both side in such scenario weak and strong respectively. Network with values like phone numbers which gives it a distinct value, between two user nodes get some additional values like time of call ‘one voi, ork edge gets weight by number of calls mode by it to other. Network : sntact each other like frequently, rarely, never get connected,9 Data Analytics (MU) 10.3 Clustering of Social Network Graphs Guster gives data into subsets of related or linked nodes of a graph tend to duster together. Graphs are used to represent data by few clustering algorithms, Clusters can be Generated on ‘based properties. 10.3.1 Distance Measure for Social-Network Graphs + Measuring @ distance is an essential task for applying clustering Cr) technique on any graph. Few graph edge has label, it represents distance measure. Some edge of graph may be unlabeled. + The distance dix, y) = 0 if there is an existence or presence of an edge ie, nodes appear close as there is an edge. The distance dix y) = 1 means no edge or nodes appear distant. @) + Land = can be used to represent values for an existing edge. Fig. 10.3.1 : Example for triangle + ‘Oand 1 or ‘A and = these are not true 2-valued distance measure. Uses of these values vio inequality when there are edge and nodes combination as shown in Fig. 10.3.1 '* In above example, there is edge (A.B) and edge (8, C) but there is no any edge between node A ie edge AQ, * Above example can be valued by assigning value 1 to distance of an existing edge and 1.5 to ¢ missing edge 10.3.2 Applying Standard Cluster Method objects. Custer coefficient gives degree to . ‘+ Clustering is popular technique to find patters from large dataset domain. It is useful in visualization | and hypothesis generation. anon | Fee, | atom [irr pao aa] ‘Cluster a Feedback loop Ser pen every 1 ee 4 Stein his,Se ae So Soe or 90 ke kee f°. hss 10 nie ° Which are part of a single and same cister.Evty node generates a cluster for clustering algorithm, nted by ‘it is an input to algorithm. ative in nature, it work on numerical data and its easy to for implementation. N Criterion (BIC) of Minimum Description Length (MDL) can be used to estimate K ‘any distance measure with K-medoids, K-medoids is general version of K-means. tive variable types and numerical. i variables and outliers Euclidean distances do not work in better way. tionally costlier. ee when data points are not avaiable (Le. pair wise10-7 Mining Social Nety W_ Big Data Analytics (MU) (0) Fuzzy C-means clustering (FCM) It is unsupervised and it always converges. «allows one piece of data which is part of two or more clusters. «It isused frequently in pattern recognition. 10.3.3 Betweenness ‘sTo find communities among social network: problems with standard clustering methods. Betweenness is shortest path available between two nodes. For example an edge (%. y) is b node a and b such that the edge (x.y) lies on shortest path between a and b. «= a.andb are two different communities where edge (x y) lies somewhere as shortest path between, ss some specialized technique are developed as t 10.3.4 The Girvan - Newman Algorithm ‘+ Itis published in 2002 by Michelle Girvan and mark Newman for : © Community detection. © Tomeasure edge - betweenness among all existing edges. © To remove edge having large valued betweenness. © To option optimized modular function, ‘+ Girvan Newman algorithm checks for edge betweenness centrality and vertex betweenness c * Vertex betweenness centrality is total number of shortest path that pass through each network. ‘+ If any ambiguity found with above (vertex betweenness centrality) then every path is adj ‘weight 1/N among all N paths between two vertices. ‘i ne * Edge betweenness centrality is number of shortest path which pass through given edge. Example : Successive delete edges of high betweenness.10-8 Mining Social Network Graph ing edges of high betweenness. ness of multiple edges of highest betweenness if ‘it may affect to graph to get separate into multiple ning of graph.Big Data Analytics (MU) 20-9. end > 0Cial Nety betweenness and again remove the edge or edges of highest b Step 2: Now, recalculate all if so, these are regions nested within larg break few existing component into smaller, repeatation of tasks by recalculating all betweenness and removing the edge or edges betweenness. 10.3.5 Using Betweenness to Find Communities 4s _ This an approach to find most shortest path within a graph which connect two vertex. «tis process of systematically removal of edges, first, process is continued till graph is broken into suitable count of connected components. edges having highest betweenness are pr Fig. 10.3.4 : Betweenness score for graph example . Between ee iad i eae a * Clearly, edge (0, G) and (8, G) has highest betweenness, so it will get removed first, it components namely {A, B, C, D) and {E,F, G, H}. “' + By keeping removal with highest betweenness next it i ie Rpslehc removal are with score 6 i.e. (E.G) and (6 ‘+ Finally graph remains as : © © ® Fig. 10.3.5 : All edges with betweenness 5 and more are remove * ‘Communities’ implies that A and C more cl - lose to each other to community (A, B, C, Dy because they hare end G ousitethecommunay en ene nity. * Similarly G is “traitor” to group (EF, G, H} and only F, G and H remain connected, 10.4 Direct Disco of Communitiesre ‘gest set of vertices where any two vertices needs to be thin @ graph is known as maximum clique vertices which can be partitioned into pnecessary of having same size. id to be bipirate if and only if it does not posses a cycle of an odd length two disjoint sets suppose set V and set U. Both V and have 5 engines and I cos a my 5 species Where each mechanic has different skills and can handle Bhich it _ eer, ee two vertices to shows that the mechanics has necessary skill to . termining maximum matchit ; a ted by workforce. matching we can maximize the number Bipirate Graph seid to be complete bipirate graph as its vertex set partitioned into two subsets of m and ely. connected if they belong to different subsets, approach of analyze a social network graphs. of various types of nodes, simrank is useful to calculate the similarity from same type nodes. ulfor random walkers on a social graph while starting with a particular node. ams calculation and itis done at every starting node for limited sizes graph. racy jIker on Social Network i. aa raph is mostly undirected and web graph founds directed, Random walker of social graph number of neighboring node of it E ple as shown in Fig. 10.5.1 U-users, T-tags, W-web pages. * » walker will o for U, or Wi, If walker prefer to go Wy then in next attempt it will iit to T, or to visit U, then in next attempt it will meet either Ty Tz or Ts z_Big Data Analytics (MU) tosis ‘+ T,and T, both tag placed on webpage W, and reachable by user U, and U> 10.5.2 Random Walks with Restart t Example : Mining Social ‘A tripartite graph example for random walker social network Fig. 10.5. network. Irres, By keeping trend of visiting start at any node traversal can visit all bee of re ‘node walker can vist all node so itis known as random walker on social network. Random node visiting walker may stop at some node in random. To know when a random walker may stop can be calculate with help of probabilities putting in a transition. ‘Suppose, M is transition matrix of graph G entering at row a column b of M is 1/k if node b of gf degree K and one of the adjacent node is a else entry is 0 (zero). Fig. 10.5.2: A simple bipartite social graph In Fig, 10.5.2 network consists of three images, and two tags *Fog" and "Grass" Imagel and Image 3 tags and image 2 has only one tag ie. "Grass", Image 1 and Image 3 are comparably more similar than Image 2 and random walker with restart at probably support that intention after applying analysis to it. wink__Mining Social Network Graph of node "Grass" which is conn }zEF0 entries to node “Grass” eo lected t Oe Lim mn oF image node. If therefore it has some degree mn must have to be 1/3, des correspond to fist three rows nd fir first three rows of column 5, areca St three columns of transaction matrix so entry 1/3 ay "fe 9" node does not have an edge to either itself or “Grass” Probability the walker will teleport to initial node N. ey is herwise its 0 (zero) Triangles using MapReduce 9 MapReduce 7 Enlist its application, wv oe) or adap 20 ‘approx 2.5 quintillion byte data increasing per de \ engine. rocessing and to generate large data sets. Map function gj sigg atsneed eaters ons AW_ Big Data Analytics (MU) 20-15 ee oe Let a graph G = (V.E) a simple undirected and unweighted graph. let, n= Iv] m= [El T(v) = set of neighbors of v {We V|Wwe E} ov = Tul Cluster co-efficient (cc(v)) for a node where v € Vis, (Seem = Hume clue toe Tl Above expression gives cluster coefficient for node v ‘* Map Reduce is used in page ranking, Its also useful in: () Web access log states, (i) Inverted index construction (ii) Document clustering {iv)_ Statistical machine translation (W) Machine learning (vi) Web link-graph reversal (vii) Distributed sorting (vil) Distributed pattern based search (x) Machine translation @.1. What is sociogram and Barbasi-Albert algorithm ? Q.2 How can a social network treated as graph ? Q.3 Explain the graph parameters listed below : () Degree (i) Geosestic distance (ill) Density: 4 How degree, closeness, between's centrally is measured ? @.5 What is social network ? Explain any one type in details. as Explain following clustering algorithm in short : (a) Hierarchical (0) Kmeans. (© Kmedoid (6) FuzzyC-means Q.7 Explain Girvan-Newman algorithm in details with help of suitable example. @.8 Whats betweenness 7 Explain use of betweenness to find communities, Q.9 Explain bpirate graph and complete biprate graph, Q.10 What is simrank ? Explain in brief. Q.11_ What is MapReduce ? Enlist its application 2