Elastic @ Adobe: Making Search Smarter with Machine Learning at Scale
2 likes11,409 views
The document outlines Adobe's innovative search capabilities that leverage machine learning and computer vision to enhance search functionalities across various media types, including non-textual content. It discusses the architecture and deployment of Elasticsearch within Adobe's ecosystem, highlighting performance metrics such as document volume and query ingestion rates. Additionally, it details advancements in image and video search through deep learned representations and clustering techniques for improving search accuracy and efficiency.
Elastic @ Adobe: Making Search Smarter with Machine Learning at Scale
- 1. © 2018 Adobe. All Rights Reserved. Adobe Confidential.© 2018 Adobe. All Rights Reserved. Adobe Confidential. Elastic @ Adobe: Making Search Smarter with Machine Learning at Scale --- Baldo Faieta & Gaurav Kukal
- 2. © 2018 Adobe. All Rights Reserved. Adobe Confidential.© 2018 Adobe. All Rights Reserved. Adobe Confidential. 2 Adobe is unique in Search space as large part of content is non textual in nature like images, videos, 3d templates, psd, dcx but at the same time billions of content pieces have text as well ….
- 3. © 2018 Adobe. All Rights Reserved. Adobe Confidential.© 2018 Adobe. All Rights Reserved. Adobe Confidential. A buffet of Search Use Cases 3 1 2 3 4 5 Search Based on Computer Vision & Metadata Deep Textual and hybrid content Search Video and Richer format Search Enterprise Search Discovery and Recommendation s
- 4. © 2018 Adobe. All Rights Reserved. Adobe Confidential.© 2018 Adobe. All Rights Reserved. Adobe Confidential. “Computer vision is an interdisciplinary field that deals with how computers can be made to gain high-level understanding from digital images or videos. From the perspective of engineering, it seeks to automate tasks that the human visual system can do” Computer Vision, ML, AI to the rescue
- 5. © 2018 Adobe. All Rights Reserved. Adobe Confidential.© 2018 Adobe. All Rights Reserved. Adobe Confidential.
- 6. © 2018 Adobe. All Rights Reserved. Adobe Confidential.© 2018 Adobe. All Rights Reserved. Adobe Confidential. Landscape of Search Product Wise 6 Adobe Experience CloudAdobe Creative Cloud Adobe Document Cloud ADOBE MARKETING CLOUD ADOBE ANALYTICS CLOUD ADOBE ADVERTISING CLOUD Experience Manager Campaign PrimetimeTarget Audience Manager Analytics
- 7. © 2018 Adobe. All Rights Reserved. Adobe Confidential.© 2018 Adobe. All Rights Reserved. Adobe Confidential. Landscape of Search Product Wise : Current State 7 Adobe Experience CloudAdobe Creative Cloud Adobe Document Cloud ADOBE MARKETING CLOUD ADOBE ANALYTICS CLOUD ADOBE ADVERTISING CLOUD Experience Manager Campaign PrimetimeTarget Audience Manager Analytics
- 8. © 2018 Adobe. All Rights Reserved. Adobe Confidential.© 2018 Adobe. All Rights Reserved. Adobe Confidential. 8 SEARCH EXPERIENCE MATTERS MORE THAN EVER
- 9. © 2018 Adobe. All Rights Reserved. Adobe Confidential.© 2018 Adobe. All Rights Reserved. Adobe Confidential. Architecture Search @Adobe Birds Eye View
- 10. © 2018 Adobe. All Rights Reserved. Adobe Confidential.© 2018 Adobe. All Rights Reserved. Adobe Confidential.
- 11. © 2018 Adobe. All Rights Reserved. Adobe Confidential.© 2018 Adobe. All Rights Reserved. Adobe Confidential.
- 12. © 2018 Adobe. All Rights Reserved. Adobe Confidential.© 2018 Adobe. All Rights Reserved. Adobe Confidential.
- 13. © 2018 Adobe. All Rights Reserved. Adobe Confidential.© 2018 Adobe. All Rights Reserved. Adobe Confidential.
- 14. © 2018 Adobe. All Rights Reserved. Adobe Confidential.© 2018 Adobe. All Rights Reserved. Adobe Confidential.
- 15. © 2018 Adobe. All Rights Reserved. Adobe Confidential.© 2018 Adobe. All Rights Reserved. Adobe Confidential.
- 16. © 2018 Adobe. All Rights Reserved. Adobe Confidential.© 2018 Adobe. All Rights Reserved. Adobe Confidential. Adobe Search Elasticsearch Stats 3 Geographical Regions ~10 Billion Documents 18 Production Elastic Search Clusters * As of June 2018 200 Shards Max # Shards 16 ~400 Virtual Machines in AWS and Azure 2 Public Clouds ~6000 Live Ingestion Rate/ second ~25000 Ingestion Rate/second Capacity for Reindexing ~600 Queries Per Second
- 17. © 2018 Adobe. All Rights Reserved. Adobe Confidential.© 2018 Adobe. All Rights Reserved. Adobe Confidential. Adobe Search Elasticsearch : Self Managed * As of June 2018 17 Self Managed Clusters on public clouds1 Moved Lr Search from AWS Elasticsearch hosted to Self- Managed, 3.4 billion docs 2
- 18. © 2018 Adobe. All Rights Reserved. Adobe Confidential.© 2018 Adobe. All Rights Reserved. Adobe Confidential. Why Elasticsearch? * As of June 2018 18 Because we have seen Solr Cloud code and cloud is an after thought 1 Stability and resilience is very high (if done right ) 2 Right balance of open-source with tight review process 3
- 19. © 2018 Adobe. All Rights Reserved. Adobe Confidential.© 2018 Adobe. All Rights Reserved. Adobe Confidential. Adobe Search Play with Elasticsearch Elasticsearch as a white box ES Plugin for Image/Video similarity & Search Ranking * As of June 2018 19 1 Cost Saving: Hot & Cold Indices & Static and Live Indices 2 Index any Generic Entity into Elastic Search with zero code change 3 Out of order Event : Optimistic locks using ES scripting 4 Custom ACLs in Elastic Search for Enterprise Searches 5
- 20. © 2018 Adobe. All Rights Reserved. Adobe Confidential.© 2018 Adobe. All Rights Reserved. Adobe Confidential. ML Deep dive
- 21. © 2018 Adobe Systems Incorporated. All Rights Reserved. Stock powered by Elasticsearch License images used in projects, ads, websites, etc. • Over 130M images • Images indexed by tags, price, type, … • Novel visual search applications to surface more results as well as differentiator Do these better using deep learned representations (DLR)
- 22. © 2018 Adobe Systems Incorporated. All Rights Reserved. Deep Learned Representation: Embedding Project to continuous space where similarity of particular property can be mapped to (Euclidean) distance: Dense, small (1k dimensions) vectors Similarity score corresponds to square distance Dimensions usually don’t have meaning Usually trained using (deep) neural networks Used to power deep search engine Image embedding space Image Similar tags (“semantic”) maps to Word (word2vec) Similar word context Face Similar face attributes
- 23. © 2018 Adobe Systems Incorporated. All Rights Reserved. Image Embedding Trained using a neural network (CNN) On an auxiliary task (e.g., classification) Layer 1 Layer 4 Layer 5 CNN Embedding Layer 1 DOG 0 SURF 1 SUNSET 0 SUN 1 WOMAN 0 PALM 0 HEAD 0 50s 1 BEACH 0 STARFISH 0 SMILE 0 RETRIEVER … … WOMAN, DOG, BEACH, … Training: • Feed image and network tries to predict tag indicators. Errors backpropagated Embedding layer = layer before last • Embedding is output of embedding layer • Captures abstract rep. of tags for image • 1k dimensions vector • Learns filters activated when patterns arise at various layers
- 24. © 2018 Adobe Systems Incorporated. All Rights Reserved. Deep Search Engine 24 Get embeddings for all images and index them Deep Search Engine Embeddings Query embedding Query Query using an embedding corresp. to query image Relevant results are those images whose embeddings are closest to query embedding using Euclidean distance Close by embeddings corresp. to ”semantically” similar Finding similar images: • Calculate similarity for all candidates • Pick top k
- 25. © 2018 Adobe Systems Incorporated. All Rights Reserved. Towards a Reverse Index for Embeddings Too expensive to calc. distance with 100M+ per query 3 1 4 8 15 18 20 14 16 17 56 7 12 9 10 13 2 19 11 14 9 10 13 query …Bucket1 Bucket2 Bucket3 … … … … So, bucketize (cluster) embeddings assigning them to one of 1k buckets Find nearest 20 buckets to query from bucket centroids Only compare query with embeddings of 20 nearest buckets (2% of corpus) Reverse index: • Bucket to image embedding
- 26. © 2018 Adobe Systems Incorporated. All Rights Reserved. PQ-Codes Embeddings are still too big to keep in index (4kb * 100M+) • Also, lots of floating point distance calculations 1011 0110 1111 0100 1010 01010010 1110 1001 1101 0111 0011 1000 1100 0001 0000 1011 0110 1111 0100 1010 01010010 1110 1001 1101 0111 0011 1000 1100 0001 0000 1011 0110 1111 0100 1010 0101 0010 1110 1001 1101 0111 0011 1000 1100 0001 0000 1011 0110 1111 0100 1010 01010010 1110 1001 1101 0111 0011 1000 1100 0001 0000 s1 s2 s63 s64 1k floats / 4KB 64 bytes … … Compress embedding using trained encoder: • Subdivide embedding space in (64) subspaces During search: • Because codes quantize sub-vectors, we can pre- calculate values dependent on quantized -centroids and bucket centroids • Distance to candidate is fast because we can leverage LUTs calculated once per query per bucket • Cluster each subspace in 256 clusters • Encode every subspace-vector of embedding with ID of nearest cluster (as byte) • PQ-Code is concatenation of subspace IDs • Store pqcode, bucketID in index tabQC – 2.ql + tl + 2.tcl Query & quantized-centroids Query & Bucket Bucket Bucket & Sub-centroids PrecalculatedPrecalculatedOnce per query per bucket Once per query
- 27. © 2018 Adobe Systems Incorporated. All Rights Reserved. Elasticsearch Pqcode Plugin 27 Has to work in conjunction with other asset data in ES So, implement deep search as a plugin and store pqcodes in ES Plugin implements comparison between query embedding and candidates pqcodes CAS analyzer outputs query embedding Reverse index used to limit search to nearest 20 buckets Calculate scores for all candidates in buckets CAS Model Encode r Embeddin g + Nearest Buckets Elastic Search ES Pqcode Plugin Encode r Images index pqcode s
- 28. © 2018 Adobe Systems Incorporated. All Rights Reserved.. Demo – Find Similar Controls
- 29. © 2018 Adobe Systems Incorporated. All Rights Reserved.. Exploration/Refinement • Majority 1, 2 word queries Numberofsearches(M) 0 15 30 45 60 Number of words / query 1 2 3 4 5 • 1 word queries • Banana, christmas, family, beach, food, flowers, car, … • Very general • Exploration • 2 word queries • happy family, wood background, doctor patient, business man,… • Still too general • Refinement by query rewrite • Both modes can be helped by way of clustering
- 30. © 2018 Adobe Systems Incorporated. All Rights Reserved. Clustering PQ Codes Use pqcodes for top 5k results to cluster using k-means:
- 31. © 2018 Adobe Systems Incorporated. All Rights Reserved. Clustering PQ Codes Use pqcodes for top 5k results to cluster using k-means: Iteratively assign pqcodes to cluster of closest centroid and recalculate centroids until convergence
- 32. © 2018 Adobe Systems Incorporated. All Rights Reserved. Clustering PQ Codes Use pqcodes for top 5k results to cluster using k-means: Iteratively assign pqcodes to cluster of closest centroid and recalculate centroids until convergence Centroids like queries and assignments like find-similar Decision to assign to a cluster uses only additions and subtractions s(y1,xi)−s(y2,xi) = tabQC1 + ql1 − tabQC2 − ql2 Distance to Centroid Y1 Distance to Centroid Y2 Once per cluster centroid Once per cluster centroid per bucket
- 33. © 2018 Adobe Systems Incorporated. All Rights Reserved.. Demo – Clustering Stock Images