Handling Unit-Length Embeddings in Inference API #122878
Description
The Inference API exposes embeddings that may already be normalized to unit length. For such models, the API defaults to dot_product similarity. However, when these embeddings are ingested into Elasticsearch, we verify their normalization and reject them if their magnitude falls outside a defined epsilon threshold.
We previously found that our epsilon value was too strict in some cases, and we've encountered another instance of this issue.
Reproduction Steps
This issue can be reproduced with the following requests:
PUT _inference/text_embedding/elastic_embed_inference_id
{
"service": "amazonbedrock",
"service_settings": {
"access_key": "XXXX",
"secret_key": "XXX",
"region": "us-east-1",
"provider": "cohere",
"model": "cohere.embed-english-v3"
}
}
PUT test
{
"mappings": {
"properties": {
"text": {
"type": "semantic_text",
"inference_id": "cohere-embeddings"
}
}
}
}
PUT test/_doc/1
{
"text": "payments to be done today for server room dressing, Volopay and t-shirt vendor."
}This results in the following failure:
"reason": "The [dot_product] similarity can only be used with unit-length vectors. Preview of invalid vector: [0.0043258667, -0.040649414, -0.09881592, -0.13244629, -0.012504578, ...]"The resulting vector:
{ 0.0043258667,-0.040649414,-0.09881592,-0.13244629,-0.012504578,-6.661415E-4,-0.0769043,0.09674072,-0.004676819,0.03970337,-0.01360321,0.042755127,0.009643555,-0.01725769,0.007511139,0.0053482056,0.02116394,0.020477295,0.06726074,0.017837524,0.014640808,0.023239136,0.036346436,-0.051971436,0.018157959,0.0317688,-1.1116266E-4,0.0016183853,0.019744873,-0.0010595322,0.025146484,-0.042938232,0.015380859,-0.008262634,0.039733887,-0.021865845,0.012237549,-4.5967102E-4,0.017730713,-0.0031147003,0.035308838,-0.022705078,-0.0073013306,-0.021453857,-0.02079773,-0.022567749,0.0064582825,-0.027145386,0.025268555,0.054473877,-0.023620605,-0.007297516,-0.030944824,0.047821045,-0.012969971,-0.043273926,-0.04586792,0.08404541,0.039611816,-0.015335083,-0.022201538,-0.014945984,0.0085372925,0.02633667,-0.010101318,0.016921997,0.027130127,0.008239746,-0.021377563,-0.017822266,0.022979736,-0.034698486,-0.021850586,0.02809143,-0.032104492,-0.00422287,0.016494751,0.014923096,0.016342163,0.027435303,0.008850098,0.008216858,0.0513916,-0.0126571655,0.011436462,0.03479004,-0.011802673,-0.019088745,9.212494E-4,-0.023132324,-0.014320374,0.043548584,-0.018341064,-3.9052963E-4,-0.028366089,-0.00497818,0.047698975,0.10046387,-0.04196167,-0.02079773,-0.051574707,0.016677856,-0.040649414,-0.026062012,0.006099701,-0.053619385,-0.037200928,0.0435791,0.07446289,0.07885742,-0.033325195,0.045166016,0.05203247,-0.015579224,-0.024765015,-0.045776367,-0.007843018,0.048034668,0.03237915,0.04547119,-0.003578186,0.031097412,5.0497055E-4,-0.026351929,0.019088745,0.013511658,-0.0158844,-0.015625,3.848076E-4,0.01625061,0.021591187,0.0131073,-0.042755127,0.0073432922,-0.028320312,0.023849487,0.02053833,-0.023910522,-0.0029144287,-0.026672363,0.07928467,-0.0042037964,0.0019521713,0.033813477,-0.028213501,0.009689331,0.018661499,0.031433105,-0.002023697,-0.027816772,-0.020599365,0.0035934448,-0.035736084,0.051635742,0.08526611,0.08099365,-0.002193451,0.057128906,-0.024978638,0.097961426,0.0077934265,-0.014472961,0.026428223,-0.02255249,-0.05380249,0.0149383545,-0.0541687,-0.008201599,0.036590576,0.01876831,-0.010147095,-0.03237915,0.070495605,0.017242432,-0.02142334,0.020187378,0.019577026,-0.01751709,0.006916046,-0.016616821,-0.03149414,-0.02154541,-0.026153564,0.011917114,-0.025344849,-0.01838684,0.0026512146,0.0016231537,0.057739258,0.0050697327,0.0065345764,-0.02456665,-0.04196167,-0.032806396,-0.0021629333,-0.0067481995,0.024429321,-0.005580902,0.01448822,0.003004074,-0.08453369,0.029449463,0.005897522,0.05130005,0.06982422,-0.015258789,0.022232056,0.017700195,0.024459839,-0.0024929047,-0.018569946,-0.042907715,0.016937256,-0.008033752,-0.008583069,0.034057617,-0.023834229,0.054779053,0.04208374,0.017608643,0.06213379,-0.04135132,0.057434082,-0.016174316,-0.04458618,0.032562256,-0.01676941,0.029693604,0.009689331,0.054595947,0.009246826,-0.047027588,-0.01020813,0.0012845993,-0.03050232,0.0022468567,0.012626648,-0.016708374,0.012817383,0.028656006,0.020401001,-0.03289795,-0.01914978,0.013427734,0.021881104,0.0011920929,0.013427734,-0.0047073364,0.027008057,0.07409668,-0.003276825,-0.0010690689,-0.0041236877,-0.034851074,0.035186768,0.029815674,-0.045654297,-0.021759033,-0.018875122,0.03149414,0.002588272,0.061157227,0.040161133,0.019592285,-0.031433105,0.044189453,-0.009407043,-0.006504059,-0.026916504,0.040863037,0.0041122437,-0.058380127,-0.010826111,0.042266846,-0.0038166046,-0.015701294,-0.047332764,0.011207581,-0.001947403,0.014595032,-0.03277588,-0.011680603,0.02557373,-0.007217407,0.018920898,0.0096588135,0.03982544,-0.025039673,0.0073280334,-0.008796692,0.001964569,0.036712646,-0.025512695,0.05026245,0.005256653,-0.014892578,0.03173828,0.10021973,-0.027023315,-0.0035514832,-0.08117676,-0.03274536,-0.01776123,-0.048217773,0.006038666,0.0141067505,0.035705566,0.016952515,0.026504517,-0.012557983,0.005844116,0.023239136,0.0077438354,0.061279297,-0.033294678,-0.06738281,-0.026473999,0.005458832,-0.09820557,-0.074645996,-0.043945312,-0.040740967,-0.021606445,-0.04345703,-0.026351929,0.033813477,-0.064697266,0.0041885376,-0.017105103,-0.01737976,0.025802612,0.02470398,0.0038986206,0.07733154,0.008926392,0.0038795471,0.025619507,0.04486084,0.029006958,-0.033996582,0.045440674,-0.069885254,-0.034484863,0.042907715,-0.02645874,-0.010223389,-0.021026611,-0.010093689,-0.019561768,-0.007572174,0.01586914,-3.9458275E-4,-0.0079574585,0.009269714,-0.011955261,0.04257202,0.013183594,-0.023651123,0.002664566,-0.044555664,-0.011421204,0.010169983,-0.027816772,-0.010032654,0.0019721985,-0.030517578,-0.0074272156,0.013595581,-0.036468506,-0.016235352,-4.813671E-4,-0.06451416,0.022445679,0.0039596558,-0.020751953,0.030273438,-0.010604858,-0.033477783,0.0069236755,-0.023834229,-0.0037021637,-0.034820557,0.044006348,-0.03387451,-0.036224365,0.06591797,-0.012863159,-0.0033798218,-0.01171875,-0.04046631,0.0046424866,0.09124756,0.0047302246,-0.05218506,0.011756897,0.012359619,-0.026229858,0.023376465,-0.008865356,-0.0095825195,-0.027755737,-0.014846802,-0.0033740997,-0.022445679,0.03366089,-0.057922363,0.019866943,-0.042877197,0.031677246,0.038146973,-0.012321472,-0.020004272,-7.662773E-4,-0.02960205,-0.011741638,0.015686035,-0.021606445,5.82695E-4,-0.06524658,0.0143966675,0.03756714,-0.024261475,2.477169E-4,0.022964478,-0.038085938,-0.023239136,0.009857178,0.03112793,-0.018798828,0.04168701,-0.011047363,-0.03152466,0.002105713,-0.026779175,-0.06585693,-0.064331055,-0.030776978,-0.012763977,0.04647827,-0.05883789,-9.3221664E-4,0.004600525,0.0043525696,-0.017456055,-0.036712646,0.014122009,0.02734375,0.0513916,-0.06732178,-0.00667572,-0.010177612,0.04522705,-0.012207031,0.02456665,0.015541077,-0.0625,0.0037822723,0.09954834,0.017166138,-0.012710571,0.030715942,-0.07562256,0.020339966,0.085510254,-0.010902405,-0.008903503,0.02619934,-0.0056381226,-0.014671326,0.011451721,-0.0014543533,0.03778076,-0.037200928,0.001159668,0.032196045,0.008995056,-0.016159058,0.012489319,-0.009170532,0.013877869,-0.01725769,0.007972717,0.0519104,0.0129852295,-0.0017786026,0.020568848,-0.018310547,0.0053901672,0.020599365,-0.029800415,-0.07519531,-0.016738892,0.025436401,-0.051727295,0.005004883,0.058685303,-0.009147644,0.045196533,0.0309906,0.017990112,-0.033843994,-0.005832672,-0.04135132,-0.06137085,0.027328491,0.029846191,-4.3964386E-4,0.013015747,0.003900528,0.025253296,-0.018859863,-0.019485474,-0.02154541,0.0035381317,-0.028335571,0.052337646,-0.026733398,0.02029419,-0.012649536,-0.03414917,0.014694214,0.03466797,-0.009605408,0.015144348,0.054779053,-0.01209259,-0.031280518,0.030395508,-0.023071289,-0.01713562,-6.585121E-4,0.0063476562,0.016952515,0.016418457,0.020065308,0.0054969788,-0.02432251,-0.034851074,-0.025680542,-0.0119018555,0.0012483597,-0.013023376,-0.020141602,0.024505615,-0.035461426,0.056640625,-0.03262329,0.06451416,0.049713135,0.0087890625,0.01361084,-0.00605011,-0.061584473,-0.033416748,-0.011207581,-0.0423584,-0.04574585,0.034362793,0.007331848,-0.037139893,-0.009941101,-0.018630981,-0.021652222,0.01033783,0.025360107,0.022094727,0.025543213,-0.025115967,-0.018081665,-0.018707275,-0.035491943,0.0028820038,-0.0057640076,-0.017623901,0.0057258606,-3.168583E-4,0.042114258,0.023834229,0.0030441284,-0.046295166,-0.0021419525,-0.013755798,0.014389038,-0.009468079,-0.0021438599,0.0647583,-0.027755737,0.008995056,-6.890297E-4,0.042419434,-0.028625488,-0.01322937,-0.037841797,-0.04977417,0.020080566,0.018157959,-0.008880615,0.031021118,0.06878662,0.028808594,0.020996094,0.016647339,0.003206253,0.0020179749,0.015464783,0.031082153,0.036865234,0.05215454,-0.018997192,0.042022705,-0.008377075,-0.025024414,0.001616478,-0.024963379,-0.0070991516,0.05911255,-0.036895752,0.020996094,0.014099121,-0.04864502,-0.002319336,-0.0040893555,1.9657612E-4,0.01626587,0.008155823,-0.036499023,-0.050048828,-0.004005432,-0.0050735474,-0.030731201,0.019226074,-0.013153076,0.032928467,-0.03387451,0.06939697,0.0031032562,0.041381836,0.0446167,-0.051483154,-0.013771057,0.028427124,0.012512207,0.022628784,-0.04437256,0.0104904175,-0.0206604,-0.016326904,0.02444458,-0.0050086975,-0.010971069,-0.0017213821,-0.02154541,0.011749268,-0.06384277,-0.008903503,-0.053619385,-0.021972656,0.058929443,-0.015045166,0.017303467,-0.008033752,0.030059814,-0.0037269592,-0.009605408,0.055480957,0.034332275,0.0057411194,0.013633728,0.014640808,0.027252197,-0.017532349,-0.019836426,0.007736206,-0.014060974,-0.025360107,0.005355835,-0.008621216,0.012359619,-0.023849487,-0.011894226,0.011550903,0.0068588257,0.033203125,-0.0069122314,-0.032318115,0.05770874,-0.02949524,-0.030166626,0.017944336,-0.020492554,-0.03186035,-0.034851074,-0.002866745,-0.033843994,0.032958984,0.023391724,-0.035980225,0.027648926,0.026138306,0.0018062592,-0.0066719055,0.023529053,-1.6510487E-4,0.0022640228,0.012016296,0.032440186,0.0073394775,-0.031921387,0.02494812,0.009918213,0.0023612976,-0.016326904,-0.01360321,-0.011978149,-0.019165039,0.022491455,0.04458618,0.016738892,0.010345459,0.0446167,-0.061340332,-0.039794922,-0.018203735,-0.029174805,0.01927185,-0.049987793,-0.030349731,-0.045440674,-0.022338867,-0.035095215,0.026000977,-0.017425537,-0.0076141357,-0.0013837814,0.0016441345,-0.044799805,0.013084412,-0.014633179,-0.042388916,0.012771606,-0.015899658,-0.0016088486,-0.0030269623,-0.02166748,-0.02154541,0.045562744,0.029953003,-0.016921997,0.010116577,0.0061531067,0.008659363,0.0015211105,-0.0054512024,0.022705078,0.037841797,-0.014038086,-0.06072998,-0.020050049,0.018508911,0.0013475418,0.01763916,0.0062217712,-0.03314209,0.010971069,-0.06878662,0.014633179,-0.0927124,-0.099853516,0.04864502,0.005844116,-0.0029277802,-0.01737976,0.052368164,0.10491943,-0.03616333,0.009849548,-0.003917694,0.013687134,0.013923645,0.028457642,-0.03515625,0.06188965,-0.06756592,0.0044403076,-0.0066337585,0.017501831,0.02130127,-0.012557983,-0.0022678375,-0.0076141357,-0.009483337,0.008041382,0.0082092285,-0.008964539,0.006855011,0.02520752,-0.011833191,-0.016159058,0.0017585754,-0.0051193237,-0.007965088,-0.059692383,-0.024719238,-0.021896362,-0.026367188,0.022537231,-0.02772522,0.013244629,-0.0068244934,0.01914978,-0.0013723373,0.020095825,0.029556274,0.0034275055,0.009895325,0.0154418945,-0.0036296844,-0.023803711,-0.0053100586,0.014968872,-0.004749298,0.017791748,0.0206604,0.021530151,0.041900635,-0.062561035,0.033233643,0.035308838,-0.0011100769,0.0066604614,0.033233643,-0.010185242,-0.036895752,0.0036849976,0.015090942,0.0016508102,-0.025680542,0.013549805,0.016357422,-0.007972717,-0.0234375,-0.069885254,0.030319214,-0.023376465,-0.003742218,0.016220093,0.010879517,-0.02355957,0.010307312,0.03768921,0.016174316,-0.01876831,-0.016220093,-4.2259693E-5,0.014381409,0.031707764,-0.0015506744,-0.014572144,0.009597778,0.030426025,-0.0129852295,0.04272461,0.03591919,0.032348633,-0.0143585205,0.008766174,0.009529114,-0.017974854,0.05218506,-0.043304443,0.017913818,-0.002588272,0.026535034,-0.020996094,-0.03753662,0.041503906,0.0038776398,0.01084137,-0.013748169,0.06121826,-0.02116394,-0.0027160645,0.052490234,0.0024719238,0.01966858,-0.0390625,0.017501831,-0.017974854,-0.010940552,0.018188477,0.0072135925,-0.06616211,-0.011558533,-0.013885498,-0.031402588,0.026153564,0.0085372925,-0.026947021,-0.012283325,-0.031433105,-0.0060768127,0.021377563,0.07348633,-0.057617188,-0.02709961,0.005958557,0.009887695,6.175041E-4,0.0043182373,-0.031555176,-0.008995056,0.025817871,-6.3943863E-4,0.012512207,-0.008346558,0.008773804,-0.04736328,-0.0029182434,0.002904892,-0.009002686,-0.024993896,0.025558472,0.01499939,-0.032562256,-0.053741455,-0.012893677,0.007286072,-0.011459351,-0.033996582,-0.0027256012,-0.02394104,-0.046295166,-0.045318604,-0.005012512,-0.012252808,-0.01927185,-0.066223145,0.055999756,-0.007221222,0.007911682,-0.0088272095,0.015434265,-0.0038776398,-0.0044555664,-0.020645142,0.028869629,0.038879395,-0.011940002,-0.028564453,-0.025314331,-0.023910522,-0.008094788,0.018463135,-0.02368164,0.03878784,0.029632568,-0.013336182,-0.0017614365,-0.010055542,-0.011444092,0.008094788,-0.029953003,0.056915283,0.057861328,-0.08526611,0.028381348,-0.033843994,-0.050231934,0.035583496,0.012367249,0.015991211,0.02130127,-0.008155823,0.015182495,0.011817932,-0.015266418,-0.015350342,-0.031555176,-0.0059318542,-0.017410278,-0.02456665,0.0048103333,-0.05255127,0.05895996,-0.028747559,0.014518738,-0.039794922,-0.0362854,-0.013870239,0.0262146,-0.03112793,-0.017242432,-0.048553467,0.01586914,-0.013664246,0.004558563,-0.0018672943,0.01890564,-0.0103302,0.03463745,0.01979065,0.03768921,-0.005558014,0.047180176,-0.019821167,0.053466797,-0.006465912,0.027069092,-0.017471313,0.03918457,-0.025756836,0.034301758,-4.105568E-4,0.0041542053,0.02229309,0.014587402,0.004562378,-0.022994995,0.013908386,-0.022918701,0.004562378,-0.02015686,-0.028305054,0.025375366 };
with a magnitude is 0.9989886 which is above the current epsilon.
Since we don't control how these vectors are normalized, there are a few possible approaches to address this:
- Override the similarity setting to
cosineinstead ofdot_product. - Increase the epsilon threshold further, potentially factoring in the number of dimensions.
- Allow users to define a custom epsilon value or disable strict normalization checks.
Would love input on which approach makes the most sense or if there are alternative solutions worth considering.
I have marked this issue as a bug since the only available mitigation for users is to override the similarity setting to cosine for the external models they use.