TorchInductor CPU Performance Dashboard

[blzheng]

Dashboard to track the performance of torchinductor on CPU.

cc @ezyang @msaroufim @wconstab @bdhirsh @anijain2305 @zou3519 @soumith @ngimel @chauhang

[blzheng]

Performance Dashboard for float32 precision -- Single-Socket Multi-threads

Executive Summary

see more

We evaluate torchinductor across three benchmark suites - torchbench, huggingface and timm. We run these experiments on ICX 8375C. Each experiment runs one iteration of forward pass. For accuracy, we check the numerical correctness of forward pass outputs by comparing with native pytorch. We measure speedup by normalizing against the performance of native pytorch. We report mean compilation latency numbers and peak memory footprint reduction ratio.

Caveats

1. Batch size has been reduced to workaround OOM errors. Work is in progress to reduce peak memory footprint.
2. Experiments do not cover dynamic shapes.
3. Experimental setup does not have optimizer.

To measure performance, compilation latency and memory footprint reduction, we remove the models that fail accuracy checks.

Passrate

+----------+------------+-------------+-------------+
| Compiler | torchbench | huggingface | timm_models |
+----------+------------+-------------+-------------+
| inductor | 84%, 43/51 | 98%, 43/44  | 87%, 53/61  |
+----------+------------+-------------+-------------+

Geometric mean speedup

+----------+------------+-------------+-------------+
| Compiler | torchbench | huggingface | timm_models |
+----------+------------+-------------+-------------+
| inductor |   1.03x    |    1.08x    |    1.03x    |
+----------+------------+-------------+-------------+

Mean compilation time (seconds)

+----------+------------+-------------+-------------+
| Compiler | torchbench | huggingface | timm_models |
+----------+------------+-------------+-------------+
| inductor |    2.23    |    4.47     |    3.75     |
+----------+------------+-------------+-------------+

Peak memory footprint compression ratio (higher is better)

+----------+------------+-------------+-------------+
| Compiler | torchbench | huggingface | timm_models |
+----------+------------+-------------+-------------+
| inductor |    0.0x    |    0.0x     |    0.0x     |
+----------+------------+-------------+-------------+

torchbench suite with float32 precision

see more

Performance speedup

+-----------------------------------+------+----------+
|               name                |  bs  | inductor |
+-----------------------------------+------+----------+
|           squeezenet1_1           |  16  |  1.4738  |
|             resnet18              |  8   |  1.1306  |
|              alexnet              | 128  |  1.1131  |
|          vision_maskrcnn          |  1   |   1.1    |
|         soft_actor_critic         | 256  |  1.0935  |
|          resnext50_32x4d          |  8   |  1.0873  |
|        shufflenet_v2_x1_0         |  64  |  1.0814  |
|             resnet50              |  32  |  1.0604  |
|        mobilenet_v3_large         |  32  |  1.053   |
|           timm_resnest            |  32  |  1.0515  |
|           mobilenet_v2            |  16  |  1.0505  |
|            densenet121            |  64  |  1.0445  |
|            mnasnet1_0             |  32  |  1.0392  |
|   pytorch_CycleGAN_and_pix2pix    |  1   |  1.0286  |
|               vgg16               |  4   |  1.0269  |
|            Super_SloMo            |  6   |  1.0227  |
|           pytorch_unet            |  1   |  1.019   |
|               dcgan               | 256  |  1.0135  |
|            timm_vovnet            |  32  |  1.0131  |
|          LearningToPaint          |  96  |  1.0075  |
|       functorch_dp_cifar10        |  64  |  1.0023  |
|      resnet50_quantized_qat       |  32  |  0.9979  |
|    mobilenet_v2_quantized_qat     |  96  |  0.9979  |
|              yolov3               |  8   |  0.995   |
|     detectron2_fcos_r_50_fpn      |  1   |  0.9871  |
|           hf_Longformer           |  1   |  0.9642  |
|            timm_regnet            |  32  |  0.963   |
|        Background_Matting         |  1   |  0.9402  |
|                drq                |  1   |  0.9262  |
|               dlrm                | 2048 |  0.9218  |
|            timm_nfnet             | 128  |  0.8948  |
|           hf_DistilBert           |  1   |  0.8699  |
|               hf_T5               |  1   |  0.8467  |
|             hf_Albert             |  1   |  0.8211  |
|              hf_Bert              |  1   |  0.8087  |
|           fastNLP_Bert            |  1   |  0.7941  |
|              hf_Bart              |  1   |  0.7719  |
|      nvidia_deeprecommender       | 256  |  0.7583  |
|              hf_GPT2              |  1   |  0.7083  |
|         timm_efficientnet         |  64  |  0.6984  |
|      timm_vision_transformer      |  8   |  0.6756  |
| attention_is_all_you_need_pytorch |  32  |  0.5568  |
|           BERT_pytorch            |  2   |  0.4027  |
|           lennard_jones           | 1000 |  0.3608  |
|             tacotron2             |  0   |   0.0    |
|        speech_transformer         |  0   |   0.0    |
|            hf_BigBird             |  0   |   0.0    |
|          pytorch_stargan          |  0   |   0.0    |
|            tts_angular            |  0   |   0.0    |
|              demucs               |  0   |   0.0    |
|            hf_Reformer            |  0   |   0.0    |
+-----------------------------------+------+----------+

Accuracy

+-----------------------------------+-----+-------------+
|               name                | bs  |  inductor   |
+-----------------------------------+-----+-------------+
|           BERT_pytorch            |  2  |    pass     |
|          resnext50_32x4d          |  2  |    pass     |
|        Background_Matting         |  1  |    pass     |
|    mobilenet_v2_quantized_qat     |  2  |    pass     |
|        mobilenet_v3_large         |  2  |    pass     |
|      nvidia_deeprecommender       |  2  |    pass     |
|   pytorch_CycleGAN_and_pix2pix    |  1  |    pass     |
|           pytorch_unet            |  2  |    pass     |
|             resnet18              |  2  |    pass     |
|             resnet50              |  2  |    pass     |
|      resnet50_quantized_qat       |  2  |    pass     |
|        shufflenet_v2_x1_0         |  2  |    pass     |
|           lennard_jones           |  2  |    pass     |
|         soft_actor_critic         | 256 |    pass     |
|           squeezenet1_1           |  2  |    pass     |
|         timm_efficientnet         |  2  |    pass     |
|            timm_nfnet             |  2  |    pass     |
|            timm_regnet            |  2  |    pass     |
|           timm_resnest            |  2  |    pass     |
|      timm_vision_transformer      |  2  |    pass     |
|            timm_vovnet            |  2  |    pass     |
|               vgg16               |  2  |    pass     |
|            mnasnet1_0             |  2  |    pass     |
|           mobilenet_v2            |  2  |    pass     |
|               hf_T5               |  2  |    pass     |
|           fastNLP_Bert            |  2  |    pass     |
|          LearningToPaint          |  2  |    pass     |
|            Super_SloMo            |  2  |    pass     |
|              alexnet              |  2  |    pass     |
| attention_is_all_you_need_pytorch |  2  |    pass     |
|               dcgan               |  2  |    pass     |
|            densenet121            |  2  |    pass     |
|     detectron2_fcos_r_50_fpn      |  2  |    pass     |
|                drq                |  1  |    pass     |
|               dlrm                |  2  |    pass     |
|       functorch_dp_cifar10        |  2  |    pass     |
|             hf_Albert             |  2  |    pass     |
|              hf_Bart              |  2  |    pass     |
|              hf_Bert              |  2  |    pass     |
|           hf_DistilBert           |  2  |    pass     |
|              hf_GPT2              |  2  |    pass     |
|           hf_Longformer           |  2  |    pass     |
|              yolov3               |  2  |    pass     |
|            hf_Reformer            |  2  | fail_to_run |
|        speech_transformer         |  2  | fail_to_run |
|              demucs               |  1  | fail_to_run |
|          pytorch_stargan          | 16  | fail_to_run |
|            hf_BigBird             |  2  | fail_to_run |
|            tts_angular            |  2  | fail_to_run |
|             tacotron2             |  0  |   0.0000    |
|          vision_maskrcnn          |  0  |   0.0000    |
+-----------------------------------+-----+-------------+

Compilation latency (sec)

+-----------------------------------+------+----------+
|               name                |  bs  | inductor |
+-----------------------------------+------+----------+
|          vision_maskrcnn          |  1   | 11.5683  |
|           hf_Longformer           |  1   |  9.2219  |
|     detectron2_fcos_r_50_fpn      |  1   |  9.1967  |
|              yolov3               |  8   |  6.8279  |
|            timm_nfnet             | 128  |  5.4197  |
|            densenet121            |  64  |  4.9745  |
|            Super_SloMo            |  6   |  4.4554  |
| attention_is_all_you_need_pytorch |  32  |  3.6369  |
|              hf_Bart              |  1   |  3.5927  |
|           fastNLP_Bert            |  1   |  3.4844  |
|           BERT_pytorch            |  2   |  3.4361  |
|               hf_T5               |  1   |  3.388   |
|              hf_Bert              |  1   |  2.9421  |
|              hf_GPT2              |  1   |  2.7631  |
|            timm_regnet            |  32  |  2.6082  |
|      timm_vision_transformer      |  8   |  2.5672  |
|             hf_Albert             |  1   |  2.408   |
|         timm_efficientnet         |  64  |  2.0037  |
|        Background_Matting         |  1   |  1.8045  |
|        shufflenet_v2_x1_0         |  64  |  1.5502  |
|            timm_vovnet            |  32  |  1.5498  |
|       functorch_dp_cifar10        |  64  |  1.4493  |
|        mobilenet_v3_large         |  32  |  1.4454  |
|           timm_resnest            |  32  |  1.2875  |
|             resnet50              |  32  |  1.271   |
|          resnext50_32x4d          |  8   |  1.2652  |
|           mobilenet_v2            |  16  |  1.1627  |
|           hf_DistilBert           |  1   |  1.1205  |
|            mnasnet1_0             |  32  |  1.1095  |
|   pytorch_CycleGAN_and_pix2pix    |  1   |  0.7235  |
|           pytorch_unet            |  1   |  0.6441  |
|               dlrm                | 2048 |  0.6379  |
|           squeezenet1_1           |  16  |  0.6199  |
|          LearningToPaint          |  96  |  0.5728  |
|             resnet18              |  8   |  0.505   |
|               vgg16               |  4   |  0.4815  |
|              alexnet              | 128  |  0.3835  |
|                drq                |  1   |  0.3087  |
|      nvidia_deeprecommender       | 256  |  0.2971  |
|           lennard_jones           | 1000 |  0.1949  |
|         soft_actor_critic         | 256  |  0.1944  |
|    mobilenet_v2_quantized_qat     |  96  |  0.1375  |
|               dcgan               | 256  |  0.1053  |
|      resnet50_quantized_qat       |  32  | -0.0596  |
|              demucs               |  0   |   nan    |
|            hf_BigBird             |  0   |   nan    |
|            hf_Reformer            |  0   |   nan    |
|          pytorch_stargan          |  0   |   nan    |
|        speech_transformer         |  0   |   nan    |
|             tacotron2             |  0   |   nan    |
|            tts_angular            |  0   |   nan    |
+-----------------------------------+------+----------+

Peak Memory Compression Ratio

+-----------------------------------+------+----------+
|               name                |  bs  | inductor |
+-----------------------------------+------+----------+
|           BERT_pytorch            |  2   |   nan    |
|        Background_Matting         |  1   |   nan    |
|          LearningToPaint          |  96  |   nan    |
|            Super_SloMo            |  6   |   nan    |
|              alexnet              | 128  |   nan    |
| attention_is_all_you_need_pytorch |  32  |   nan    |
|               dcgan               | 256  |   nan    |
|              demucs               |  0   |   nan    |
|            densenet121            |  64  |   nan    |
|     detectron2_fcos_r_50_fpn      |  1   |   nan    |
|               dlrm                | 2048 |   nan    |
|                drq                |  1   |   nan    |
|           fastNLP_Bert            |  1   |   nan    |
|       functorch_dp_cifar10        |  64  |   nan    |
|             hf_Albert             |  1   |   nan    |
|              hf_Bart              |  1   |   nan    |
|              hf_Bert              |  1   |   nan    |
|            hf_BigBird             |  0   |   nan    |
|           hf_DistilBert           |  1   |   nan    |
|              hf_GPT2              |  1   |   nan    |
|           hf_Longformer           |  1   |   nan    |
|            hf_Reformer            |  0   |   nan    |
|               hf_T5               |  1   |   nan    |
|           lennard_jones           | 1000 |   nan    |
|            mnasnet1_0             |  32  |   nan    |
|           mobilenet_v2            |  16  |   nan    |
|    mobilenet_v2_quantized_qat     |  96  |   nan    |
|        mobilenet_v3_large         |  32  |   nan    |
|      nvidia_deeprecommender       | 256  |   nan    |
|   pytorch_CycleGAN_and_pix2pix    |  1   |   nan    |
|          pytorch_stargan          |  0   |   nan    |
|           pytorch_unet            |  1   |   nan    |
|             resnet18              |  8   |   nan    |
|             resnet50              |  32  |   nan    |
|      resnet50_quantized_qat       |  32  |   nan    |
|          resnext50_32x4d          |  8   |   nan    |
|        shufflenet_v2_x1_0         |  64  |   nan    |
|         soft_actor_critic         | 256  |   nan    |
|        speech_transformer         |  0   |   nan    |
|           squeezenet1_1           |  16  |   nan    |
|             tacotron2             |  0   |   nan    |
|         timm_efficientnet         |  64  |   nan    |
|            timm_nfnet             | 128  |   nan    |
|            timm_regnet            |  32  |   nan    |
|           timm_resnest            |  32  |   nan    |
|      timm_vision_transformer      |  8   |   nan    |
|            timm_vovnet            |  32  |   nan    |
|            tts_angular            |  0   |   nan    |
|               vgg16               |  4   |   nan    |
|          vision_maskrcnn          |  1   |   nan    |
|              yolov3               |  8   |   nan    |
+-----------------------------------+------+----------+

huggingface suite with float32 precision

see more

Performance speedup

+-----------------------------------------+----+----------+
|                  name                   | bs | inductor |
+-----------------------------------------+----+----------+
|            XLNetLMHeadModel             | 4  |  6.6101  |
|               GoogleFnet                | 1  |  1.2581  |
|     M2M100ForConditionalGeneration      | 2  |  1.2183  |
|             OPTForCausalLM              | 4  |  1.2159  |
|             XGLMForCausalLM             | 1  |  1.1916  |
|          MobileBertForMaskedLM          | 16 |  1.1915  |
|               DistillGPT2               | 1  |  1.1535  |
|           ElectraForCausalLM            | 1  |  1.1114  |
|          AllenaiLongformerBase          | 1  |  1.0992  |
|            YituTechConvBert             | 1  |  1.0688  |
|           RobertaForCausalLM            | 4  |  1.0421  |
|       MT5ForConditionalGeneration       | 2  |  1.0358  |
|     MobileBertForQuestionAnswering      | 32 |  1.0298  |
|         MegatronBertForCausalLM         | 2  |  1.0162  |
|           DebertaForMaskedLM            | 4  |  1.006   |
|            AlbertForMaskedLM            | 2  |  1.0018  |
|       AlbertForQuestionAnswering        | 2  |  0.9888  |
|                CamemBert                | 1  |  0.9838  |
|       DebertaForQuestionAnswering       | 4  |  0.9677  |
|            TrOCRForCausalLM             | 8  |  0.9655  |
|           PegasusForCausalLM            | 8  |  0.9586  |
|            PLBartForCausalLM            | 16 |  0.9538  |
|          DistilBertForMaskedLM          | 16 |  0.9396  |
|     PLBartForConditionalGeneration      | 8  |  0.9342  |
|     PegasusForConditionalGeneration     | 4  |  0.9337  |
|            MBartForCausalLM             | 16 |  0.908   |
|      MBartForConditionalGeneration      | 8  |  0.8992  |
|         Speech2Text2ForCausalLM         | 64 |  0.8708  |
|        BertForQuestionAnswering         | 64 |  0.8658  |
|       RobertaForQuestionAnswering       | 64 |  0.8653  |
|             BertForMaskedLM             | 64 |  0.8638  |
|     DistilBertForQuestionAnswering      | 32 |  0.8587  |
|    MegatronBertForQuestionAnswering     | 8  |  0.8549  |
|                 T5Small                 | 1  |   0.83   |
|    LayoutLMForSequenceClassification    | 16 |  0.8279  |
|           LayoutLMForMaskedLM           | 16 |  0.8273  |
| BlenderbotSmallForConditionalGeneration | 32 |  0.7867  |
|       BlenderbotSmallForCausalLM        | 64 |  0.7799  |
|             BartForCausalLM             | 2  |  0.7795  |
|       ElectraForQuestionAnswering       | 64 |  0.7742  |
|      GPT2ForSequenceClassification      | 4  |  0.7712  |
|       T5ForConditionalGeneration        | 4  |  0.7252  |
|      BartForConditionalGeneration       | 1  |  0.7104  |
|                 BigBird                 | 0  |   0.0    |
+-----------------------------------------+----+----------+

Accuracy

+-----------------------------------------+----+-------------+
|                  name                   | bs |  inductor   |
+-----------------------------------------+----+-------------+
|            AlbertForMaskedLM            | 1  |    pass     |
|       AlbertForQuestionAnswering        | 1  |    pass     |
|      MBartForConditionalGeneration      | 1  |    pass     |
|       MT5ForConditionalGeneration       | 1  |    pass     |
|         MegatronBertForCausalLM         | 1  |    pass     |
|    MegatronBertForQuestionAnswering     | 1  |    pass     |
|          MobileBertForMaskedLM          | 1  |    pass     |
|     MobileBertForQuestionAnswering      | 1  |    pass     |
|             OPTForCausalLM              | 1  |    pass     |
|            PLBartForCausalLM            | 1  |    pass     |
|     PLBartForConditionalGeneration      | 1  |    pass     |
|           PegasusForCausalLM            | 1  |    pass     |
|     PegasusForConditionalGeneration     | 1  |    pass     |
|           RobertaForCausalLM            | 1  |    pass     |
|       RobertaForQuestionAnswering       | 1  |    pass     |
|         Speech2Text2ForCausalLM         | 1  |    pass     |
|       T5ForConditionalGeneration        | 1  |    pass     |
|                 T5Small                 | 1  |    pass     |
|            TrOCRForCausalLM             | 1  |    pass     |
|             XGLMForCausalLM             | 1  |    pass     |
|            XLNetLMHeadModel             | 1  |    pass     |
|            MBartForCausalLM             | 1  |    pass     |
|     M2M100ForConditionalGeneration      | 1  |    pass     |
|    LayoutLMForSequenceClassification    | 1  |    pass     |
|           LayoutLMForMaskedLM           | 1  |    pass     |
|          AllenaiLongformerBase          | 1  |    pass     |
|             BartForCausalLM             | 1  |    pass     |
|      BartForConditionalGeneration       | 1  |    pass     |
|             BertForMaskedLM             | 1  |    pass     |
|        BertForQuestionAnswering         | 1  |    pass     |
|       BlenderbotSmallForCausalLM        | 1  |    pass     |
| BlenderbotSmallForConditionalGeneration | 1  |    pass     |
|                CamemBert                | 1  |    pass     |
|           DebertaForMaskedLM            | 1  |    pass     |
|       DebertaForQuestionAnswering       | 1  |    pass     |
|          DistilBertForMaskedLM          | 1  |    pass     |
|     DistilBertForQuestionAnswering      | 1  |    pass     |
|               DistillGPT2               | 1  |    pass     |
|           ElectraForCausalLM            | 1  |    pass     |
|       ElectraForQuestionAnswering       | 1  |    pass     |
|      GPT2ForSequenceClassification      | 1  |    pass     |
|               GoogleFnet                | 1  |    pass     |
|            YituTechConvBert             | 1  |    pass     |
|                 BigBird                 | 1  | fail_to_run |
+-----------------------------------------+----+-------------+

Compilation latency (sec)

+-----------------------------------------+----+----------+
|                  name                   | bs | inductor |
+-----------------------------------------+----+----------+
|          MobileBertForMaskedLM          | 16 | 10.4985  |
|     MobileBertForQuestionAnswering      | 32 | 10.4742  |
|          AllenaiLongformerBase          | 1  | 10.1465  |
|      BartForConditionalGeneration       | 1  |  8.0126  |
|      MBartForConditionalGeneration      | 8  |  7.766   |
|           DebertaForMaskedLM            | 4  |  7.4744  |
|       DebertaForQuestionAnswering       | 4  |  7.3231  |
|     PegasusForConditionalGeneration     | 4  |  7.2929  |
|    MegatronBertForQuestionAnswering     | 8  |  6.6289  |
|         MegatronBertForCausalLM         | 2  |  6.5296  |
|     M2M100ForConditionalGeneration      | 2  |  6.517   |
|             XGLMForCausalLM             | 1  |  5.7003  |
| BlenderbotSmallForConditionalGeneration | 32 |  5.2596  |
|            YituTechConvBert             | 1  |  4.5927  |
|       T5ForConditionalGeneration        | 4  |  4.5008  |
|           LayoutLMForMaskedLM           | 16 |  4.074   |
|       ElectraForQuestionAnswering       | 64 |  4.0555  |
|    LayoutLMForSequenceClassification    | 16 |  3.867   |
|     PLBartForConditionalGeneration      | 8  |  3.8398  |
|             BertForMaskedLM             | 64 |  3.6523  |
|       RobertaForQuestionAnswering       | 64 |  3.5445  |
|                 T5Small                 | 1  |  3.5072  |
|             BartForCausalLM             | 2  |  3.3817  |
|        BertForQuestionAnswering         | 64 |  3.3744  |
|           RobertaForCausalLM            | 4  |  3.3594  |
|      GPT2ForSequenceClassification      | 4  |  3.3011  |
|       MT5ForConditionalGeneration       | 2  |  3.277   |
|            MBartForCausalLM             | 16 |  3.1166  |
|           PegasusForCausalLM            | 8  |  3.0688  |
|                CamemBert                | 1  |  3.0637  |
|           ElectraForCausalLM            | 1  |  3.0246  |
|            TrOCRForCausalLM             | 8  |  2.9159  |
|             OPTForCausalLM              | 4  |  2.8233  |
|       AlbertForQuestionAnswering        | 2  |  2.7299  |
|            AlbertForMaskedLM            | 2  |  2.654   |
|       BlenderbotSmallForCausalLM        | 64 |  2.312   |
|               GoogleFnet                | 1  |  1.8488  |
|          DistilBertForMaskedLM          | 16 |  1.725   |
|     DistilBertForQuestionAnswering      | 32 |  1.6647  |
|            PLBartForCausalLM            | 16 |  1.6417  |
|         Speech2Text2ForCausalLM         | 64 |  1.5846  |
|               DistillGPT2               | 1  |  1.4312  |
|            XLNetLMHeadModel             | 4  | -17.1104 |
|                 BigBird                 | 0  |   nan    |
+-----------------------------------------+----+----------+

Peak Memory Compression Ratio

+-----------------------------------------+----+----------+
|                  name                   | bs | inductor |
+-----------------------------------------+----+----------+
|            AlbertForMaskedLM            | 2  |   nan    |
|       AlbertForQuestionAnswering        | 2  |   nan    |
|          AllenaiLongformerBase          | 1  |   nan    |
|             BartForCausalLM             | 2  |   nan    |
|      BartForConditionalGeneration       | 1  |   nan    |
|             BertForMaskedLM             | 64 |   nan    |
|        BertForQuestionAnswering         | 64 |   nan    |
|                 BigBird                 | 0  |   nan    |
|       BlenderbotSmallForCausalLM        | 64 |   nan    |
| BlenderbotSmallForConditionalGeneration | 32 |   nan    |
|                CamemBert                | 1  |   nan    |
|           DebertaForMaskedLM            | 4  |   nan    |
|       DebertaForQuestionAnswering       | 4  |   nan    |
|          DistilBertForMaskedLM          | 16 |   nan    |
|     DistilBertForQuestionAnswering      | 32 |   nan    |
|               DistillGPT2               | 1  |   nan    |
|           ElectraForCausalLM            | 1  |   nan    |
|       ElectraForQuestionAnswering       | 64 |   nan    |
|      GPT2ForSequenceClassification      | 4  |   nan    |
|               GoogleFnet                | 1  |   nan    |
|           LayoutLMForMaskedLM           | 16 |   nan    |
|    LayoutLMForSequenceClassification    | 16 |   nan    |
|     M2M100ForConditionalGeneration      | 2  |   nan    |
|            MBartForCausalLM             | 16 |   nan    |
|      MBartForConditionalGeneration      | 8  |   nan    |
|       MT5ForConditionalGeneration       | 2  |   nan    |
|         MegatronBertForCausalLM         | 2  |   nan    |
|    MegatronBertForQuestionAnswering     | 8  |   nan    |
|          MobileBertForMaskedLM          | 16 |   nan    |
|     MobileBertForQuestionAnswering      | 32 |   nan    |
|             OPTForCausalLM              | 4  |   nan    |
|            PLBartForCausalLM            | 16 |   nan    |
|     PLBartForConditionalGeneration      | 8  |   nan    |
|           PegasusForCausalLM            | 8  |   nan    |
|     PegasusForConditionalGeneration     | 4  |   nan    |
|           RobertaForCausalLM            | 4  |   nan    |
|       RobertaForQuestionAnswering       | 64 |   nan    |
|         Speech2Text2ForCausalLM         | 64 |   nan    |
|       T5ForConditionalGeneration        | 4  |   nan    |
|                 T5Small                 | 1  |   nan    |
|            TrOCRForCausalLM             | 8  |   nan    |
|             XGLMForCausalLM             | 1  |   nan    |
|            XLNetLMHeadModel             | 4  |   nan    |
|            YituTechConvBert             | 1  |   nan    |
+-----------------------------------------+----+----------+

timm_models suite with float32 precision

see more

Performance speedup

+---------------------------------+-----+----------+
|              name               | bs  | inductor |
+---------------------------------+-----+----------+
|          pnasnet5large          | 16  |  1.4097  |
|          inception_v3           | 128 |  1.1948  |
|       gluon_inception_v3        | 128 |  1.1843  |
|        adv_inception_v3         | 128 |  1.1781  |
|        res2net101_26w_4s        | 64  |  1.0904  |
|             dpn107              | 32  |  1.0665  |
|             dla102              | 64  |  1.0544  |
|            repvgg_a2            | 128 |  1.0423  |
|            hrnet_w18            |  2  |  1.0366  |
|        res2net50_14w_8s         |  2  |  1.0365  |
|            gernet_l             | 128 |  1.036   |
|            lcnet_050            | 128 |  1.0349  |
|        ese_vovnet19b_dw         | 128 |  1.0294  |
|      mobilenetv3_large_100      | 128 |  1.0176  |
|         mobilenetv2_100         | 128 |  1.0172  |
|           fbnetc_100            | 128 |  1.0149  |
|           resnest101e           | 32  |  1.0148  |
|          gmixer_24_224          | 64  |  1.0147  |
|          cspdarknet53           | 64  |  1.0142  |
|           mnasnet_100           | 128 |  1.0138  |
|          ghostnet_100           | 128 |  1.0114  |
|          spnasnet_100           | 128 |  1.0066  |
|     swsl_resnext101_32x16d      | 32  |  1.0063  |
|        convmixer_768_32         | 32  |  1.0046  |
|            fbnetv3_b            | 128 |  1.0024  |
|        gluon_xception65         | 32  |  1.0008  |
|           selecsls42b           | 128 |  0.9956  |
|           regnety_002           | 128 |  0.9546  |
|           res2next50            |  2  |  0.9453  |
|         poolformer_m36          | 64  |  0.9104  |
|            nfnet_l0             | 64  |  0.9095  |
|           dm_nfnet_f0           | 128 |  0.9038  |
|      xcit_large_24_p8_224       |  5  |  0.8902  |
|           tf_mixnet_l           | 64  |  0.8771  |
|           volo_d1_224           | 64  |  0.8742  |
|  swin_base_patch4_window7_224   | 64  |  0.8543  |
|      beit_base_patch16_224      | 64  |  0.8432  |
|            mixnet_l             | 64  |  0.8371  |
|           rexnet_100            | 128 |  0.8289  |
|          mixer_b16_224          | 64  |  0.8269  |
|          cait_m36_384           |  2  |  0.8253  |
|          resmlp_12_224          | 128 |  0.8249  |
| deit_base_distilled_patch16_224 | 64  |  0.8189  |
|           convit_base           | 32  |  0.8189  |
|      vit_base_patch16_224       | 64  |  0.8177  |
|         visformer_small         | 128 |  0.8131  |
|        tnt_s_patch16_224        | 64  |  0.7996  |
|          convnext_base          | 32  |  0.7934  |
|         coat_lite_mini          | 128 |  0.7662  |
|       tf_efficientnet_b0        | 128 |  0.7497  |
|           mobilevit_s           | 32  |  0.7402  |
|          jx_nest_base           | 32  |  0.732   |
|        twins_pcpvt_base         | 32  |  0.7223  |
|            pit_b_224            | 64  |  0.7186  |
|            tinynet_a            | 128 |  0.701   |
|         crossvit_9_240          | 64  |  0.6647  |
|          gmlp_s16_224           | 64  |  0.6182  |
|          botnet26t_256          |  0  |   0.0    |
|        eca_halonext26ts         |  0  |   0.0    |
|       eca_botnext26ts_256       |  0  |   0.0    |
|        sebotnet33ts_256         |  0  |   0.0    |
+---------------------------------+-----+----------+

Accuracy

+---------------------------------+----+---------------+
|              name               | bs |   inductor    |
+---------------------------------+----+---------------+
|        adv_inception_v3         | 2  |     pass      |
|          resmlp_12_224          | 2  |     pass      |
|         mobilenetv2_100         | 2  |     pass      |
|      mobilenetv3_large_100      | 2  |     pass      |
|           mobilevit_s           | 2  |     pass      |
|            nfnet_l0             | 2  |     pass      |
|            pit_b_224            | 2  |     pass      |
|          pnasnet5large          | 2  |     pass      |
|         poolformer_m36          | 2  |     pass      |
|           regnety_002           | 2  |     pass      |
|            repvgg_a2            | 2  |     pass      |
|        res2net101_26w_4s        | 2  |     pass      |
|        res2net50_14w_8s         | 2  |     pass      |
|           res2next50            | 2  |     pass      |
|           resnest101e           | 2  |     pass      |
|      beit_base_patch16_224      | 2  |     pass      |
|           rexnet_100            | 2  |     pass      |
|           selecsls42b           | 2  |     pass      |
|          spnasnet_100           | 2  |     pass      |
|     swsl_resnext101_32x16d      | 2  |     pass      |
|       tf_efficientnet_b0        | 2  |     pass      |
|           tf_mixnet_l           | 2  |     pass      |
|            tinynet_a            | 2  |     pass      |
|        tnt_s_patch16_224        | 2  |     pass      |
|        twins_pcpvt_base         | 2  |     pass      |
|         visformer_small         | 2  |     pass      |
|      vit_base_patch16_224       | 2  |     pass      |
|           volo_d1_224           | 2  |     pass      |
|           mnasnet_100           | 2  |     pass      |
|            mixnet_l             | 2  |     pass      |
|          mixer_b16_224          | 2  |     pass      |
|        ese_vovnet19b_dw         | 2  |     pass      |
|         coat_lite_mini          | 2  |     pass      |
|           convit_base           | 2  |     pass      |
|        convmixer_768_32         | 2  |     pass      |
|          convnext_base          | 2  |     pass      |
|         crossvit_9_240          | 2  |     pass      |
|          cspdarknet53           | 2  |     pass      |
| deit_base_distilled_patch16_224 | 2  |     pass      |
|             dla102              | 2  |     pass      |
|           dm_nfnet_f0           | 2  |     pass      |
|            lcnet_050            | 2  |     pass      |
|      xcit_large_24_p8_224       | 2  |     pass      |
|           fbnetc_100            | 2  |     pass      |
|       gluon_inception_v3        | 2  |     pass      |
|          jx_nest_base           | 2  |     pass      |
|          inception_v3           | 2  |     pass      |
|            hrnet_w18            | 2  |     pass      |
|          gmlp_s16_224           | 2  |     pass      |
|          gmixer_24_224          | 2  |     pass      |
|        gluon_xception65         | 2  |     pass      |
|            gernet_l             | 2  |     pass      |
|            fbnetv3_b            | 2  |     pass      |
|        eca_halonext26ts         | 2  |  fail_to_run  |
|        sebotnet33ts_256         | 2  |  fail_to_run  |
|          botnet26t_256          | 2  |  fail_to_run  |
|       eca_botnext26ts_256       | 2  |  fail_to_run  |
|  swin_base_patch4_window7_224   | 2  | fail_accuracy |
|          ghostnet_100           | 2  | fail_accuracy |
|             dpn107              | 2  | fail_accuracy |
|          cait_m36_384           | 2  | fail_accuracy |
+---------------------------------+----+---------------+

Compilation latency (sec)

+---------------------------------+-----+----------+
|              name               | bs  | inductor |
+---------------------------------+-----+----------+
|  swin_base_patch4_window7_224   | 64  | 10.7024  |
|          cait_m36_384           |  2  | 10.2615  |
|      xcit_large_24_p8_224       |  5  |  9.9294  |
|         poolformer_m36          | 64  |  8.9775  |
|          jx_nest_base           | 32  |  8.5136  |
|            hrnet_w18            |  2  |  8.3372  |
|        twins_pcpvt_base         | 32  |  8.1557  |
|          pnasnet5large          | 16  |  6.6776  |
|        tnt_s_patch16_224        | 64  |  6.528   |
|          gmlp_s16_224           | 64  |  6.4682  |
|           tf_mixnet_l           | 64  |  6.137   |
|           volo_d1_224           | 64  |  6.1065  |
|           mobilevit_s           | 32  |  6.0721  |
|           dm_nfnet_f0           | 128 |  5.7608  |
|            mixnet_l             | 64  |  5.7029  |
|         crossvit_9_240          | 64  |  4.9915  |
|          convnext_base          | 32  |  4.9459  |
|        res2net101_26w_4s        | 64  |  4.7408  |
|            pit_b_224            | 64  |  4.6434  |
|             dpn107              | 32  |  4.4485  |
|         coat_lite_mini          | 128 |  4.308   |
|        res2net50_14w_8s         |  2  |  4.2332  |
|            fbnetv3_b            | 128 |  3.9204  |
|           convit_base           | 32  |  3.9035  |
|      beit_base_patch16_224      | 64  |  3.7148  |
|          gmixer_24_224          | 64  |  3.0949  |
| deit_base_distilled_patch16_224 | 64  |  3.0859  |
|         visformer_small         | 128 |  3.0531  |
|            nfnet_l0             | 64  |  2.9973  |
|            tinynet_a            | 128 |  2.9481  |
|      vit_base_patch16_224       | 64  |  2.888   |
|       tf_efficientnet_b0        | 128 |  2.8524  |
|           rexnet_100            | 128 |  2.8057  |
|          mixer_b16_224          | 64  |  2.6488  |
|           res2next50            |  2  |  2.4974  |
|          cspdarknet53           | 64  |  2.362   |
|          ghostnet_100           | 128 |  2.1989  |
|       gluon_inception_v3        | 128 |  2.1866  |
|        gluon_xception65         | 32  |  2.0955  |
|             dla102              | 64  |   2.09   |
|           fbnetc_100            | 128 |  1.8692  |
|          spnasnet_100           | 128 |  1.8467  |
|           regnety_002           | 128 |  1.8316  |
|          resmlp_12_224          | 128 |  1.8239  |
|        convmixer_768_32         | 32  |  1.8029  |
|            gernet_l             | 128 |  1.718   |
|        adv_inception_v3         | 128 |  1.704   |
|          inception_v3           | 128 |  1.6711  |
|      mobilenetv3_large_100      | 128 |  1.6126  |
|            repvgg_a2            | 128 |  1.5339  |
|           mnasnet_100           | 128 |  1.5139  |
|        ese_vovnet19b_dw         | 128 |  1.3457  |
|         mobilenetv2_100         | 128 |  1.2205  |
|     swsl_resnext101_32x16d      | 32  |  1.1662  |
|           selecsls42b           | 128 |  0.9562  |
|            lcnet_050            | 128 |  0.9299  |
|           resnest101e           | 32  | -1.6045  |
|          botnet26t_256          |  0  |   nan    |
|       eca_botnext26ts_256       |  0  |   nan    |
|        eca_halonext26ts         |  0  |   nan    |
|        sebotnet33ts_256         |  0  |   nan    |
+---------------------------------+-----+----------+

Peak Memory Compression Ratio

+---------------------------------+-----+----------+
|              name               | bs  | inductor |
+---------------------------------+-----+----------+
|        adv_inception_v3         | 128 |   nan    |
|      beit_base_patch16_224      | 64  |   nan    |
|          botnet26t_256          |  0  |   nan    |
|          cait_m36_384           |  2  |   nan    |
|         coat_lite_mini          | 128 |   nan    |
|           convit_base           | 32  |   nan    |
|        convmixer_768_32         | 32  |   nan    |
|          convnext_base          | 32  |   nan    |
|         crossvit_9_240          | 64  |   nan    |
|          cspdarknet53           | 64  |   nan    |
| deit_base_distilled_patch16_224 | 64  |   nan    |
|             dla102              | 64  |   nan    |
|           dm_nfnet_f0           | 128 |   nan    |
|             dpn107              | 32  |   nan    |
|       eca_botnext26ts_256       |  0  |   nan    |
|        eca_halonext26ts         |  0  |   nan    |
|        ese_vovnet19b_dw         | 128 |   nan    |
|           fbnetc_100            | 128 |   nan    |
|            fbnetv3_b            | 128 |   nan    |
|            gernet_l             | 128 |   nan    |
|          ghostnet_100           | 128 |   nan    |
|       gluon_inception_v3        | 128 |   nan    |
|        gluon_xception65         | 32  |   nan    |
|          gmixer_24_224          | 64  |   nan    |
|          gmlp_s16_224           | 64  |   nan    |
|            hrnet_w18            |  2  |   nan    |
|          inception_v3           | 128 |   nan    |
|          jx_nest_base           | 32  |   nan    |
|            lcnet_050            | 128 |   nan    |
|          mixer_b16_224          | 64  |   nan    |
|            mixnet_l             | 64  |   nan    |
|           mnasnet_100           | 128 |   nan    |
|         mobilenetv2_100         | 128 |   nan    |
|      mobilenetv3_large_100      | 128 |   nan    |
|           mobilevit_s           | 32  |   nan    |
|            nfnet_l0             | 64  |   nan    |
|            pit_b_224            | 64  |   nan    |
|          pnasnet5large          | 16  |   nan    |
|         poolformer_m36          | 64  |   nan    |
|           regnety_002           | 128 |   nan    |
|            repvgg_a2            | 128 |   nan    |
|        res2net101_26w_4s        | 64  |   nan    |
|        res2net50_14w_8s         |  2  |   nan    |
|           res2next50            |  2  |   nan    |
|          resmlp_12_224          | 128 |   nan    |
|           resnest101e           | 32  |   nan    |
|           rexnet_100            | 128 |   nan    |
|        sebotnet33ts_256         |  0  |   nan    |
|           selecsls42b           | 128 |   nan    |
|          spnasnet_100           | 128 |   nan    |
|  swin_base_patch4_window7_224   | 64  |   nan    |
|     swsl_resnext101_32x16d      | 32  |   nan    |
|       tf_efficientnet_b0        | 128 |   nan    |
|           tf_mixnet_l           | 64  |   nan    |
|            tinynet_a            | 128 |   nan    |
|        tnt_s_patch16_224        | 64  |   nan    |
|        twins_pcpvt_base         | 32  |   nan    |
|         visformer_small         | 128 |   nan    |
|      vit_base_patch16_224       | 64  |   nan    |
|           volo_d1_224           | 64  |   nan    |
|      xcit_large_24_p8_224       |  5  |   nan    |
+---------------------------------+-----+----------+

[blzheng]

Performance Dashboard for float32 precision -- Single-core Single-thread

Executive Summary

see more

We evaluate torchinductor across three benchmark suites - torchbench, huggingface and timm. We run these experiments on Intel(R) Xeon(R) Platinum 8358 CPU @ 2.60GHz. Each experiment runs one iteration of forward pass. For accuracy, we check the numerical correctness of forward pass outputs by comparing with native pytorch. We measure speedup by normalizing against the performance of native pytorch. We report mean compilation latency numbers and peak memory footprint reduction ratio.

Caveats

1. Batch size has been reduced to workaround OOM errors. Work is in progress to reduce peak memory footprint.
2. Experiments do not cover dynamic shapes.
3. Experimental setup does not have optimizer.

To measure performance, compilation latency and memory footprint reduction, we remove the models that fail accuracy checks.

Passrate

+----------+------------+-------------+-------------+
| Compiler | torchbench | huggingface | timm_models |
+----------+------------+-------------+-------------+
| inductor | 87%, 47/54 | 93%, 41/44  | 87%, 53/61  |
+----------+------------+-------------+-------------+

Geometric mean speedup

+----------+------------+-------------+-------------+
| Compiler | torchbench | huggingface | timm_models |
+----------+------------+-------------+-------------+
| inductor |   1.05x    |    1.00x    |    1.04x    |
+----------+------------+-------------+-------------+

Mean compilation time (seconds)

+----------+------------+-------------+-------------+
| Compiler | torchbench | huggingface | timm_models |
+----------+------------+-------------+-------------+
| inductor |   17.67    |    21.57    |    37.83    |
+----------+------------+-------------+-------------+

Peak memory footprint compression ratio (higher is better)

+----------+------------+-------------+-------------+
| Compiler | torchbench | huggingface | timm_models |
+----------+------------+-------------+-------------+
| inductor |    0.0x    |    0.0x     |    0.0x     |
+----------+------------+-------------+-------------+

torchbench suite with float32 precision

see more

Performance speedup

+-----------------------------------+------+----------+
|               name                |  bs  | inductor |
+-----------------------------------+------+----------+
|           squeezenet1_1           |  16  |  2.0426  |
|       functorch_dp_cifar10        |  64  |  1.3211  |
|        shufflenet_v2_x1_0         |  64  |  1.2811  |
|           timm_resnest            |  32  |  1.2005  |
|              alexnet              | 128  |  1.1775  |
|             resnet18              |  8   |  1.1458  |
|          vision_maskrcnn          |  1   |  1.0896  |
|               vgg16               |  4   |  1.0833  |
|            densenet121            |  64  |  1.0796  |
|             resnet50              |  32  |  1.0712  |
|          resnext50_32x4d          |  8   |  1.0589  |
|            timm_vovnet            |  32  |  1.0577  |
|            Super_SloMo            |  6   |  1.053   |
|                drq                |  1   |  1.0514  |
|              yolov3               |  8   |  1.0338  |
|   pytorch_CycleGAN_and_pix2pix    |  1   |  1.0336  |
|         soft_actor_critic         | 256  |  1.0149  |
|          LearningToPaint          |  96  |  1.0088  |
|            timm_regnet            |  32  |  1.004   |
|           pytorch_unet            |  1   |  1.0029  |
| attention_is_all_you_need_pytorch |  32  |  1.0016  |
|               dlrm                | 2048 |  1.0014  |
|      resnet50_quantized_qat       |  32  |  1.0002  |
|    mobilenet_v2_quantized_qat     |  96  |   1.0    |
|               dcgan               | 256  |  0.9906  |
|        mobilenet_v3_large         |  32  |  0.9839  |
|          pytorch_stargan          |  16  |  0.9812  |
|           mobilenet_v2            |  16  |  0.9542  |
|            mnasnet1_0             |  32  |  0.9527  |
|        Background_Matting         |  1   |  0.951   |
|   timm_vision_transformer_large   |  8   |  0.7949  |
|           BERT_pytorch            |  2   |  0.7655  |
|           hf_Longformer           |  1   |  0.7648  |
|              hf_GPT2              |  1   |  0.7537  |
|            timm_nfnet             | 128  |  0.7182  |
|      nvidia_deeprecommender       | 256  |  0.6899  |
|           hf_DistilBert           |  1   |  0.6658  |
|             hf_Albert             |  1   |  0.6541  |
|            hf_T5_large            |  1   |  0.6501  |
|              hf_Bert              |  1   |  0.6301  |
|           fastNLP_Bert            |  1   |  0.5986  |
|              hf_Bart              |  1   |  0.5983  |
|           hf_GPT2_large           |  1   |  0.5679  |
|      timm_vision_transformer      |  8   |  0.5591  |
|               hf_T5               |  1   |  0.5566  |
|            hf_T5_base             |  1   |  0.4761  |
|         timm_efficientnet         |  64  |  0.4639  |
|           lennard_jones           | 1000 |  0.284   |
|            hf_BigBird             |  0   |   0.0    |
|            hf_Reformer            |  0   |   0.0    |
|            tts_angular            |  0   |   0.0    |
|     detectron2_fcos_r_50_fpn      |  0   |   0.0    |
|              demucs               |  0   |   0.0    |
|        speech_transformer         |  0   |   0.0    |
+-----------------------------------+------+----------+

Accuracy

+-----------------------------------+-----+-------------+
|               name                | bs  |  inductor   |
+-----------------------------------+-----+-------------+
|           BERT_pytorch            |  2  |    pass     |
|          resnext50_32x4d          |  2  |    pass     |
|           mobilenet_v2            |  2  |    pass     |
|    mobilenet_v2_quantized_qat     |  2  |    pass     |
|        mobilenet_v3_large         |  2  |    pass     |
|      nvidia_deeprecommender       |  2  |    pass     |
|   pytorch_CycleGAN_and_pix2pix    |  1  |    pass     |
|          pytorch_stargan          | 16  |    pass     |
|           pytorch_unet            |  2  |    pass     |
|             resnet18              |  2  |    pass     |
|             resnet50              |  2  |    pass     |
|      resnet50_quantized_qat       |  2  |    pass     |
|        shufflenet_v2_x1_0         |  2  |    pass     |
|           lennard_jones           |  2  |    pass     |
|         soft_actor_critic         | 256 |    pass     |
|           squeezenet1_1           |  2  |    pass     |
|         timm_efficientnet         |  2  |    pass     |
|            timm_nfnet             |  2  |    pass     |
|            timm_regnet            |  2  |    pass     |
|           timm_resnest            |  2  |    pass     |
|      timm_vision_transformer      |  2  |    pass     |
|   timm_vision_transformer_large   |  2  |    pass     |
|            timm_vovnet            |  2  |    pass     |
|               vgg16               |  2  |    pass     |
|        Background_Matting         |  1  |    pass     |
|            mnasnet1_0             |  2  |    pass     |
|            hf_T5_large            |  2  |    pass     |
|       functorch_dp_cifar10        |  2  |    pass     |
|          LearningToPaint          |  2  |    pass     |
|            Super_SloMo            |  2  |    pass     |
|              alexnet              |  2  |    pass     |
| attention_is_all_you_need_pytorch |  2  |    pass     |
|               dcgan               |  2  |    pass     |
|            densenet121            |  2  |    pass     |
|               dlrm                |  2  |    pass     |
|            hf_T5_base             |  2  |    pass     |
|           fastNLP_Bert            |  2  |    pass     |
|                drq                |  1  |    pass     |
|             hf_Albert             |  2  |    pass     |
|              hf_Bart              |  2  |    pass     |
|              hf_Bert              |  2  |    pass     |
|           hf_DistilBert           |  2  |    pass     |
|              hf_GPT2              |  2  |    pass     |
|           hf_GPT2_large           |  2  |    pass     |
|           hf_Longformer           |  2  |    pass     |
|               hf_T5               |  2  |    pass     |
|              yolov3               |  2  |    pass     |
|            hf_BigBird             |  2  | fail_to_run |
|     detectron2_fcos_r_50_fpn      |  2  | fail_to_run |
|              demucs               |  1  | fail_to_run |
|            hf_Reformer            |  2  | fail_to_run |
|            tts_angular            |  2  | fail_to_run |
|        speech_transformer         |  0  |   0.0000    |
|          vision_maskrcnn          |  0  |   0.0000    |
+-----------------------------------+-----+-------------+

Compilation latency (sec)

+-----------------------------------+------+----------+
|               name                |  bs  | inductor |
+-----------------------------------+------+----------+
|            hf_T5_base             |  1   | 110.8172 |
|            timm_nfnet             | 128  | 89.3988  |
|            densenet121            |  64  | 83.6673  |
|           hf_GPT2_large           |  1   | 68.8979  |
|   timm_vision_transformer_large   |  8   | 54.2422  |
|            hf_T5_large            |  1   | 49.4562  |
|         timm_efficientnet         |  64  | 39.8036  |
|          vision_maskrcnn          |  1   | 39.3907  |
|            Super_SloMo            |  6   | 34.6477  |
|        Background_Matting         |  1   | 26.0698  |
|              yolov3               |  8   | 25.8651  |
|           pytorch_unet            |  1   | 20.5692  |
|        mobilenet_v3_large         |  32  |  18.621  |
|            timm_regnet            |  32  | 16.8904  |
|      timm_vision_transformer      |  8   | 13.7887  |
|           hf_Longformer           |  1   | 12.8186  |
|        shufflenet_v2_x1_0         |  64  | 11.4008  |
|              hf_Bart              |  1   | 10.0706  |
|           mobilenet_v2            |  16  |  9.776   |
|            timm_vovnet            |  32  |  9.7671  |
|   pytorch_CycleGAN_and_pix2pix    |  1   |  9.2664  |
|          LearningToPaint          |  96  |  9.0346  |
|          pytorch_stargan          |  16  |  8.9297  |
|               hf_T5               |  1   |  8.3078  |
|           fastNLP_Bert            |  1   |  7.3867  |
|           timm_resnest            |  32  |  7.3225  |
|               vgg16               |  4   |  7.1152  |
|             resnet50              |  32  |  6.4957  |
|            mnasnet1_0             |  32  |  6.1132  |
| attention_is_all_you_need_pytorch |  32  |  5.6499  |
|           squeezenet1_1           |  16  |  5.4609  |
|             hf_Albert             |  1   |   5.4    |
|              hf_Bert              |  1   |  5.222   |
|              hf_GPT2              |  1   |  4.8789  |
|              alexnet              | 128  |  4.5239  |
|           BERT_pytorch            |  2   |  4.098   |
|           hf_DistilBert           |  1   |  3.285   |
|             resnet18              |  8   |  2.9366  |
|               dcgan               | 256  |  2.7928  |
|      nvidia_deeprecommender       | 256  |  2.6756  |
|       functorch_dp_cifar10        |  64  |  2.4487  |
|          resnext50_32x4d          |  8   |  1.7993  |
|               dlrm                | 2048 |  1.6011  |
|                drq                |  1   |  0.4278  |
|         soft_actor_critic         | 256  |  0.2726  |
|           lennard_jones           | 1000 |  0.2698  |
|    mobilenet_v2_quantized_qat     |  96  |  0.1607  |
|      resnet50_quantized_qat       |  32  |  0.1281  |
|              demucs               |  0   |   nan    |
|     detectron2_fcos_r_50_fpn      |  0   |   nan    |
|            hf_BigBird             |  0   |   nan    |
|            hf_Reformer            |  0   |   nan    |
|        speech_transformer         |  0   |   nan    |
|            tts_angular            |  0   |   nan    |
+-----------------------------------+------+----------+

Peak Memory Compression Ratio

+-----------------------------------+------+----------+
|               name                |  bs  | inductor |
+-----------------------------------+------+----------+
|           BERT_pytorch            |  2   |   nan    |
|        Background_Matting         |  1   |   nan    |
|          LearningToPaint          |  96  |   nan    |
|            Super_SloMo            |  6   |   nan    |
|              alexnet              | 128  |   nan    |
| attention_is_all_you_need_pytorch |  32  |   nan    |
|               dcgan               | 256  |   nan    |
|              demucs               |  0   |   nan    |
|            densenet121            |  64  |   nan    |
|     detectron2_fcos_r_50_fpn      |  0   |   nan    |
|               dlrm                | 2048 |   nan    |
|                drq                |  1   |   nan    |
|           fastNLP_Bert            |  1   |   nan    |
|       functorch_dp_cifar10        |  64  |   nan    |
|             hf_Albert             |  1   |   nan    |
|              hf_Bart              |  1   |   nan    |
|              hf_Bert              |  1   |   nan    |
|            hf_BigBird             |  0   |   nan    |
|           hf_DistilBert           |  1   |   nan    |
|              hf_GPT2              |  1   |   nan    |
|           hf_GPT2_large           |  1   |   nan    |
|           hf_Longformer           |  1   |   nan    |
|            hf_Reformer            |  0   |   nan    |
|               hf_T5               |  1   |   nan    |
|            hf_T5_base             |  1   |   nan    |
|            hf_T5_large            |  1   |   nan    |
|           lennard_jones           | 1000 |   nan    |
|            mnasnet1_0             |  32  |   nan    |
|           mobilenet_v2            |  16  |   nan    |
|    mobilenet_v2_quantized_qat     |  96  |   nan    |
|        mobilenet_v3_large         |  32  |   nan    |
|      nvidia_deeprecommender       | 256  |   nan    |
|   pytorch_CycleGAN_and_pix2pix    |  1   |   nan    |
|          pytorch_stargan          |  16  |   nan    |
|           pytorch_unet            |  1   |   nan    |
|             resnet18              |  8   |   nan    |
|             resnet50              |  32  |   nan    |
|      resnet50_quantized_qat       |  32  |   nan    |
|          resnext50_32x4d          |  8   |   nan    |
|        shufflenet_v2_x1_0         |  64  |   nan    |
|         soft_actor_critic         | 256  |   nan    |
|        speech_transformer         |  0   |   nan    |
|           squeezenet1_1           |  16  |   nan    |
|         timm_efficientnet         |  64  |   nan    |
|            timm_nfnet             | 128  |   nan    |
|            timm_regnet            |  32  |   nan    |
|           timm_resnest            |  32  |   nan    |
|      timm_vision_transformer      |  8   |   nan    |
|   timm_vision_transformer_large   |  8   |   nan    |
|            timm_vovnet            |  32  |   nan    |
|            tts_angular            |  0   |   nan    |
|               vgg16               |  4   |   nan    |
|          vision_maskrcnn          |  1   |   nan    |
|              yolov3               |  8   |   nan    |
+-----------------------------------+------+----------+

huggingface suite with float32 precision

see more

Performance speedup

+-----------------------------------------+----+----------+
|                  name                   | bs | inductor |
+-----------------------------------------+----+----------+
|            XLNetLMHeadModel             | 4  |  0.9522  |
|       AlbertForQuestionAnswering        | 2  |  0.8971  |
|            AlbertForMaskedLM            | 2  |  0.8924  |
|       DebertaForQuestionAnswering       | 4  |  0.8817  |
|     MobileBertForQuestionAnswering      | 32 |  0.8509  |
|     M2M100ForConditionalGeneration      | 2  |  0.8381  |
|               GoogleFnet                | 1  |  0.8183  |
|           DebertaForMaskedLM            | 4  |  0.8167  |
|             OPTForCausalLM              | 4  |  0.8164  |
|            YituTechConvBert             | 1  |  0.7934  |
|             XGLMForCausalLM             | 1  |  0.7806  |
|    MegatronBertForQuestionAnswering     | 8  |  0.7733  |
|     PegasusForConditionalGeneration     | 4  |  0.7671  |
|      MBartForConditionalGeneration      | 8  |   0.76   |
|            TrOCRForCausalLM             | 8  |  0.7563  |
|           PegasusForCausalLM            | 8  |  0.7555  |
|            MBartForCausalLM             | 16 |  0.7508  |
|          MobileBertForMaskedLM          | 16 |  0.7499  |
|          AllenaiLongformerBase          | 1  |  0.7458  |
|       RobertaForQuestionAnswering       | 64 |  0.739   |
|     DistilBertForQuestionAnswering      | 32 |  0.7389  |
|        BertForQuestionAnswering         | 64 |  0.7387  |
|           RobertaForCausalLM            | 4  |  0.7317  |
|     PLBartForConditionalGeneration      | 8  |  0.7185  |
|             BertForMaskedLM             | 64 |  0.7183  |
|          DistilBertForMaskedLM          | 16 |  0.7094  |
|            PLBartForCausalLM            | 16 |  0.6941  |
|         Speech2Text2ForCausalLM         | 64 |  0.6843  |
|    LayoutLMForSequenceClassification    | 16 |  0.6161  |
|               DistillGPT2               | 1  |  0.6031  |
|                CamemBert                | 1  |  0.602   |
|           LayoutLMForMaskedLM           | 16 |  0.5932  |
| BlenderbotSmallForConditionalGeneration | 32 |  0.5907  |
|       BlenderbotSmallForCausalLM        | 64 |  0.5839  |
|             BartForCausalLM             | 2  |  0.5709  |
|      BartForConditionalGeneration       | 1  |  0.5091  |
|      GPT2ForSequenceClassification      | 4  |  0.493   |
|       ElectraForQuestionAnswering       | 64 |   0.47   |
|       T5ForConditionalGeneration        | 4  |  0.4668  |
|                 T5Small                 | 1  |  0.4642  |
|           ElectraForCausalLM            | 1  |   0.43   |
|         MegatronBertForCausalLM         | 0  |   0.0    |
|                 BigBird                 | 0  |   0.0    |
|       MT5ForConditionalGeneration       | 0  |   0.0    |
+-----------------------------------------+----+----------+

Accuracy

+-----------------------------------------+----+-------------+
|                  name                   | bs |  inductor   |
+-----------------------------------------+----+-------------+
|            AlbertForMaskedLM            | 1  |    pass     |
|       AlbertForQuestionAnswering        | 1  |    pass     |
|      MBartForConditionalGeneration      | 1  |    pass     |
|       MT5ForConditionalGeneration       | 1  |    pass     |
|         MegatronBertForCausalLM         | 1  |    pass     |
|    MegatronBertForQuestionAnswering     | 1  |    pass     |
|          MobileBertForMaskedLM          | 1  |    pass     |
|     MobileBertForQuestionAnswering      | 1  |    pass     |
|             OPTForCausalLM              | 1  |    pass     |
|            PLBartForCausalLM            | 1  |    pass     |
|     PLBartForConditionalGeneration      | 1  |    pass     |
|           PegasusForCausalLM            | 1  |    pass     |
|     PegasusForConditionalGeneration     | 1  |    pass     |
|           RobertaForCausalLM            | 1  |    pass     |
|       RobertaForQuestionAnswering       | 1  |    pass     |
|         Speech2Text2ForCausalLM         | 1  |    pass     |
|       T5ForConditionalGeneration        | 1  |    pass     |
|                 T5Small                 | 1  |    pass     |
|            TrOCRForCausalLM             | 1  |    pass     |
|             XGLMForCausalLM             | 1  |    pass     |
|            XLNetLMHeadModel             | 1  |    pass     |
|            MBartForCausalLM             | 1  |    pass     |
|     M2M100ForConditionalGeneration      | 1  |    pass     |
|    LayoutLMForSequenceClassification    | 1  |    pass     |
|           LayoutLMForMaskedLM           | 1  |    pass     |
|          AllenaiLongformerBase          | 1  |    pass     |
|             BartForCausalLM             | 1  |    pass     |
|      BartForConditionalGeneration       | 1  |    pass     |
|             BertForMaskedLM             | 1  |    pass     |
|        BertForQuestionAnswering         | 1  |    pass     |
|       BlenderbotSmallForCausalLM        | 1  |    pass     |
| BlenderbotSmallForConditionalGeneration | 1  |    pass     |
|                CamemBert                | 1  |    pass     |
|           DebertaForMaskedLM            | 1  |    pass     |
|       DebertaForQuestionAnswering       | 1  |    pass     |
|          DistilBertForMaskedLM          | 1  |    pass     |
|     DistilBertForQuestionAnswering      | 1  |    pass     |
|               DistillGPT2               | 1  |    pass     |
|           ElectraForCausalLM            | 1  |    pass     |
|       ElectraForQuestionAnswering       | 1  |    pass     |
|      GPT2ForSequenceClassification      | 1  |    pass     |
|               GoogleFnet                | 1  |    pass     |
|            YituTechConvBert             | 1  |    pass     |
|                 BigBird                 | 1  | fail_to_run |
+-----------------------------------------+----+-------------+

Compilation latency (sec)

+-----------------------------------------+----+----------+
|                  name                   | bs | inductor |
+-----------------------------------------+----+----------+
|       T5ForConditionalGeneration        | 4  | 55.5113  |
|       ElectraForQuestionAnswering       | 64 | 53.9419  |
|      GPT2ForSequenceClassification      | 4  |  51.708  |
|      BartForConditionalGeneration       | 1  | 48.6713  |
|           LayoutLMForMaskedLM           | 16 | 47.4153  |
|    LayoutLMForSequenceClassification    | 16 | 36.5231  |
|             BartForCausalLM             | 2  | 34.1458  |
|             BertForMaskedLM             | 64 | 33.9874  |
| BlenderbotSmallForConditionalGeneration | 32 | 30.9125  |
|       BlenderbotSmallForCausalLM        | 64 | 30.2863  |
|            AlbertForMaskedLM            | 2  | 26.5338  |
|           DebertaForMaskedLM            | 4  | 26.4973  |
|     PegasusForConditionalGeneration     | 4  | 26.3414  |
|                 T5Small                 | 1  | 25.6895  |
|      MBartForConditionalGeneration      | 8  | 24.7751  |
|            XLNetLMHeadModel             | 4  | 24.4838  |
|          AllenaiLongformerBase          | 1  | 22.8838  |
|     MobileBertForQuestionAnswering      | 32 |  20.116  |
|       AlbertForQuestionAnswering        | 2  | 19.5679  |
|       RobertaForQuestionAnswering       | 64 | 19.4393  |
|    MegatronBertForQuestionAnswering     | 8  | 18.6799  |
|        BertForQuestionAnswering         | 64 | 18.6026  |
|            MBartForCausalLM             | 16 |  18.417  |
|          MobileBertForMaskedLM          | 16 | 16.5618  |
|       DebertaForQuestionAnswering       | 4  | 15.1709  |
|     PLBartForConditionalGeneration      | 8  | 13.9216  |
|         Speech2Text2ForCausalLM         | 64 | 12.1095  |
|           PegasusForCausalLM            | 8  | 11.5466  |
|          DistilBertForMaskedLM          | 16 | 11.5357  |
|     DistilBertForQuestionAnswering      | 32 | 11.0201  |
|     M2M100ForConditionalGeneration      | 2  | 10.8537  |
|            PLBartForCausalLM            | 16 | 10.2818  |
|            YituTechConvBert             | 1  |  8.115   |
|             XGLMForCausalLM             | 1  |  8.0807  |
|            TrOCRForCausalLM             | 8  |  7.8229  |
|                CamemBert                | 1  |  6.4793  |
|           RobertaForCausalLM            | 4  |  6.2543  |
|           ElectraForCausalLM            | 1  |  5.9684  |
|             OPTForCausalLM              | 4  |  5.4036  |
|               DistillGPT2               | 1  |  5.024   |
|               GoogleFnet                | 1  |  3.2717  |
|                 BigBird                 | 0  |   nan    |
|       MT5ForConditionalGeneration       | 0  |   nan    |
|         MegatronBertForCausalLM         | 0  |   nan    |
+-----------------------------------------+----+----------+

Peak Memory Compression Ratio

+-----------------------------------------+----+----------+
|                  name                   | bs | inductor |
+-----------------------------------------+----+----------+
|            AlbertForMaskedLM            | 2  |   nan    |
|       AlbertForQuestionAnswering        | 2  |   nan    |
|          AllenaiLongformerBase          | 1  |   nan    |
|             BartForCausalLM             | 2  |   nan    |
|      BartForConditionalGeneration       | 1  |   nan    |
|             BertForMaskedLM             | 64 |   nan    |
|        BertForQuestionAnswering         | 64 |   nan    |
|                 BigBird                 | 0  |   nan    |
|       BlenderbotSmallForCausalLM        | 64 |   nan    |
| BlenderbotSmallForConditionalGeneration | 32 |   nan    |
|                CamemBert                | 1  |   nan    |
|           DebertaForMaskedLM            | 4  |   nan    |
|       DebertaForQuestionAnswering       | 4  |   nan    |
|          DistilBertForMaskedLM          | 16 |   nan    |
|     DistilBertForQuestionAnswering      | 32 |   nan    |
|               DistillGPT2               | 1  |   nan    |
|           ElectraForCausalLM            | 1  |   nan    |
|       ElectraForQuestionAnswering       | 64 |   nan    |
|      GPT2ForSequenceClassification      | 4  |   nan    |
|               GoogleFnet                | 1  |   nan    |
|           LayoutLMForMaskedLM           | 16 |   nan    |
|    LayoutLMForSequenceClassification    | 16 |   nan    |
|     M2M100ForConditionalGeneration      | 2  |   nan    |
|            MBartForCausalLM             | 16 |   nan    |
|      MBartForConditionalGeneration      | 8  |   nan    |
|       MT5ForConditionalGeneration       | 0  |   nan    |
|         MegatronBertForCausalLM         | 0  |   nan    |
|    MegatronBertForQuestionAnswering     | 8  |   nan    |
|          MobileBertForMaskedLM          | 16 |   nan    |
|     MobileBertForQuestionAnswering      | 32 |   nan    |
|             OPTForCausalLM              | 4  |   nan    |
|            PLBartForCausalLM            | 16 |   nan    |
|     PLBartForConditionalGeneration      | 8  |   nan    |
|           PegasusForCausalLM            | 8  |   nan    |
|     PegasusForConditionalGeneration     | 4  |   nan    |
|           RobertaForCausalLM            | 4  |   nan    |
|       RobertaForQuestionAnswering       | 64 |   nan    |
|         Speech2Text2ForCausalLM         | 64 |   nan    |
|       T5ForConditionalGeneration        | 4  |   nan    |
|                 T5Small                 | 1  |   nan    |
|            TrOCRForCausalLM             | 8  |   nan    |
|             XGLMForCausalLM             | 1  |   nan    |
|            XLNetLMHeadModel             | 4  |   nan    |
|            YituTechConvBert             | 1  |   nan    |
+-----------------------------------------+----+----------+

timm_models suite with float32 precision

see more

Performance speedup

+---------------------------------+-----+----------+
|              name               | bs  | inductor |
+---------------------------------+-----+----------+
|          pnasnet5large          | 16  |  1.664   |
|          inception_v3           | 128 |  1.3505  |
|       gluon_inception_v3        | 128 |  1.3323  |
|        adv_inception_v3         | 128 |  1.3165  |
|           res2next50            |  2  |  1.1219  |
|        res2net50_14w_8s         |  2  |  1.0979  |
|        ese_vovnet19b_dw         | 128 |  1.0866  |
|             dla102              | 64  |  1.0829  |
|           resnest101e           | 32  |  1.0724  |
|        res2net101_26w_4s        | 64  |  1.0501  |
|            hrnet_w18            |  2  |  1.0403  |
|             dpn107              | 32  |  1.0259  |
|            repvgg_a2            | 128 |  1.0171  |
|        convmixer_768_32         | 32  |  1.0058  |
|            gernet_l             | 128 |  0.998   |
|     swsl_resnext101_32x16d      | 32  |  0.9949  |
|          ghostnet_100           | 128 |  0.9917  |
|          cspdarknet53           | 64  |  0.9902  |
|           selecsls42b           | 128 |  0.9854  |
|        gluon_xception65         | 32  |  0.9847  |
|            lcnet_050            | 128 |  0.983   |
|           regnety_002           | 128 |  0.979   |
|            fbnetv3_b            | 128 |  0.9709  |
|      mobilenetv3_large_100      | 128 |  0.9617  |
|          spnasnet_100           | 128 |  0.9466  |
|           mnasnet_100           | 128 |  0.9465  |
|           fbnetc_100            | 128 |  0.9433  |
|         mobilenetv2_100         | 128 |  0.942   |
|          gmixer_24_224          | 64  |  0.8434  |
|      xcit_large_24_p8_224       |  5  |  0.835   |
|            nfnet_l0             | 64  |  0.7418  |
|         poolformer_m36          | 64  |  0.7288  |
|          cait_m36_384           |  2  |  0.7162  |
|           dm_nfnet_f0           | 128 |  0.6992  |
| deit_base_distilled_patch16_224 | 64  |  0.6954  |
|          mixer_b16_224          | 64  |  0.6919  |
|      beit_base_patch16_224      | 64  |  0.6911  |
|      vit_base_patch16_224       | 64  |  0.6855  |
|  swin_base_patch4_window7_224   | 64  |  0.6759  |
|           convit_base           | 32  |  0.6559  |
|          resmlp_12_224          | 128 |  0.6513  |
|           tf_mixnet_l           | 64  |  0.633   |
|            mixnet_l             | 64  |  0.6284  |
|           volo_d1_224           | 64  |  0.6235  |
|         visformer_small         | 128 |  0.6217  |
|          convnext_base          | 32  |  0.6019  |
|           rexnet_100            | 128 |  0.5803  |
|        tnt_s_patch16_224        | 64  |  0.561   |
|            pit_b_224            | 64  |  0.5376  |
|        twins_pcpvt_base         | 32  |  0.5318  |
|          jx_nest_base           | 32  |  0.5297  |
|         coat_lite_mini          | 128 |  0.5291  |
|         crossvit_9_240          | 64  |  0.4988  |
|       tf_efficientnet_b0        | 128 |  0.4821  |
|           mobilevit_s           | 32  |   0.47   |
|            tinynet_a            | 128 |  0.4677  |
|          gmlp_s16_224           | 64  |  0.419   |
|          botnet26t_256          |  0  |   0.0    |
|        eca_halonext26ts         |  0  |   0.0    |
|        sebotnet33ts_256         |  0  |   0.0    |
|       eca_botnext26ts_256       |  0  |   0.0    |
+---------------------------------+-----+----------+

Accuracy

+---------------------------------+----+---------------+
|              name               | bs |   inductor    |
+---------------------------------+----+---------------+
|        adv_inception_v3         | 2  |     pass      |
|          resmlp_12_224          | 2  |     pass      |
|         mobilenetv2_100         | 2  |     pass      |
|      mobilenetv3_large_100      | 2  |     pass      |
|           mobilevit_s           | 2  |     pass      |
|            nfnet_l0             | 2  |     pass      |
|            pit_b_224            | 2  |     pass      |
|          pnasnet5large          | 2  |     pass      |
|         poolformer_m36          | 2  |     pass      |
|           regnety_002           | 2  |     pass      |
|            repvgg_a2            | 2  |     pass      |
|        res2net101_26w_4s        | 2  |     pass      |
|        res2net50_14w_8s         | 2  |     pass      |
|           res2next50            | 2  |     pass      |
|           resnest101e           | 2  |     pass      |
|      beit_base_patch16_224      | 2  |     pass      |
|           rexnet_100            | 2  |     pass      |
|           selecsls42b           | 2  |     pass      |
|          spnasnet_100           | 2  |     pass      |
|     swsl_resnext101_32x16d      | 2  |     pass      |
|       tf_efficientnet_b0        | 2  |     pass      |
|           tf_mixnet_l           | 2  |     pass      |
|            tinynet_a            | 2  |     pass      |
|        tnt_s_patch16_224        | 2  |     pass      |
|        twins_pcpvt_base         | 2  |     pass      |
|         visformer_small         | 2  |     pass      |
|      vit_base_patch16_224       | 2  |     pass      |
|           volo_d1_224           | 2  |     pass      |
|           mnasnet_100           | 2  |     pass      |
|            mixnet_l             | 2  |     pass      |
|          mixer_b16_224          | 2  |     pass      |
|        ese_vovnet19b_dw         | 2  |     pass      |
|         coat_lite_mini          | 2  |     pass      |
|           convit_base           | 2  |     pass      |
|        convmixer_768_32         | 2  |     pass      |
|          convnext_base          | 2  |     pass      |
|         crossvit_9_240          | 2  |     pass      |
|          cspdarknet53           | 2  |     pass      |
| deit_base_distilled_patch16_224 | 2  |     pass      |
|             dla102              | 2  |     pass      |
|           dm_nfnet_f0           | 2  |     pass      |
|            lcnet_050            | 2  |     pass      |
|      xcit_large_24_p8_224       | 2  |     pass      |
|           fbnetc_100            | 2  |     pass      |
|       gluon_inception_v3        | 2  |     pass      |
|          jx_nest_base           | 2  |     pass      |
|          inception_v3           | 2  |     pass      |
|            hrnet_w18            | 2  |     pass      |
|          gmlp_s16_224           | 2  |     pass      |
|          gmixer_24_224          | 2  |     pass      |
|        gluon_xception65         | 2  |     pass      |
|            gernet_l             | 2  |     pass      |
|            fbnetv3_b            | 2  |     pass      |
|        eca_halonext26ts         | 2  |  fail_to_run  |
|        sebotnet33ts_256         | 2  |  fail_to_run  |
|          botnet26t_256          | 2  |  fail_to_run  |
|       eca_botnext26ts_256       | 2  |  fail_to_run  |
|  swin_base_patch4_window7_224   | 2  | fail_accuracy |
|          ghostnet_100           | 2  | fail_accuracy |
|             dpn107              | 2  | fail_accuracy |
|          cait_m36_384           | 2  | fail_accuracy |
+---------------------------------+----+---------------+

Compilation latency (sec)

+---------------------------------+-----+-----------+
|              name               | bs  | inductor  |
+---------------------------------+-----+-----------+
|           dm_nfnet_f0           | 128 | 120.4526  |
|            pit_b_224            | 64  |  93.2877  |
|         coat_lite_mini          | 128 |  86.3203  |
|  swin_base_patch4_window7_224   | 64  |   85.21   |
|           rexnet_100            | 128 |  78.2162  |
|          jx_nest_base           | 32  |  73.2018  |
|        twins_pcpvt_base         | 32  |  73.0605  |
|             dpn107              | 32  |  68.3539  |
|            nfnet_l0             | 64  |  59.3309  |
|           volo_d1_224           | 64  |   59.28   |
|          gmlp_s16_224           | 64  |  54.9854  |
|         poolformer_m36          | 64  |  54.1717  |
|        tnt_s_patch16_224        | 64  |  53.9854  |
|          pnasnet5large          | 16  |  53.3107  |
|       tf_efficientnet_b0        | 128 |  52.2643  |
|            tinynet_a            | 128 |  51.7527  |
|         visformer_small         | 128 |  49.5635  |
|          convnext_base          | 32  |  48.0006  |
|           mobilevit_s           | 32  |  47.8097  |
|           convit_base           | 32  |  47.6997  |
|            mixnet_l             | 64  |  47.2449  |
| deit_base_distilled_patch16_224 | 64  |  47.1471  |
|      vit_base_patch16_224       | 64  |  46.3161  |
|         crossvit_9_240          | 64  |  45.886   |
|      beit_base_patch16_224      | 64  |  43.2621  |
|          cait_m36_384           |  2  |  40.3293  |
|      xcit_large_24_p8_224       |  5  |  39.4825  |
|            fbnetv3_b            | 128 |  39.2445  |
|          mixer_b16_224          | 64  |  36.5714  |
|             dla102              | 64  |  34.2857  |
|          ghostnet_100           | 128 |  32.7922  |
|           tf_mixnet_l           | 64  |  29.7102  |
|        res2net101_26w_4s        | 64  |  29.1843  |
|          cspdarknet53           | 64  |  27.721   |
|          resmlp_12_224          | 128 |  24.0163  |
|        gluon_xception65         | 32  |  22.7874  |
|        ese_vovnet19b_dw         | 128 |  21.0097  |
|          gmixer_24_224          | 64  |  19.8152  |
|        adv_inception_v3         | 128 |  18.1021  |
|           fbnetc_100            | 128 |  15.4501  |
|           selecsls42b           | 128 |  14.6314  |
|      mobilenetv3_large_100      | 128 |  13.3493  |
|         mobilenetv2_100         | 128 |  13.0842  |
|           regnety_002           | 128 |  12.2577  |
|            gernet_l             | 128 |  11.3181  |
|            repvgg_a2            | 128 |  8.4116   |
|            lcnet_050            | 128 |  7.8323   |
|            hrnet_w18            |  2  |  7.7945   |
|        convmixer_768_32         | 32  |  5.9951   |
|        res2net50_14w_8s         |  2  |  5.1672   |
|           mnasnet_100           | 128 |  4.6516   |
|          spnasnet_100           | 128 |  3.0418   |
|           res2next50            |  2  |  2.0724   |
|       gluon_inception_v3        | 128 | -17.0452  |
|          inception_v3           | 128 | -18.3518  |
|     swsl_resnext101_32x16d      | 32  | -26.2001  |
|           resnest101e           | 32  | -132.8819 |
|          botnet26t_256          |  0  |    nan    |
|       eca_botnext26ts_256       |  0  |    nan    |
|        eca_halonext26ts         |  0  |    nan    |
|        sebotnet33ts_256         |  0  |    nan    |
+---------------------------------+-----+-----------+

Peak Memory Compression Ratio

+---------------------------------+-----+----------+
|              name               | bs  | inductor |
+---------------------------------+-----+----------+
|        adv_inception_v3         | 128 |   nan    |
|      beit_base_patch16_224      | 64  |   nan    |
|          botnet26t_256          |  0  |   nan    |
|          cait_m36_384           |  2  |   nan    |
|         coat_lite_mini          | 128 |   nan    |
|           convit_base           | 32  |   nan    |
|        convmixer_768_32         | 32  |   nan    |
|          convnext_base          | 32  |   nan    |
|         crossvit_9_240          | 64  |   nan    |
|          cspdarknet53           | 64  |   nan    |
| deit_base_distilled_patch16_224 | 64  |   nan    |
|             dla102              | 64  |   nan    |
|           dm_nfnet_f0           | 128 |   nan    |
|             dpn107              | 32  |   nan    |
|       eca_botnext26ts_256       |  0  |   nan    |
|        eca_halonext26ts         |  0  |   nan    |
|        ese_vovnet19b_dw         | 128 |   nan    |
|           fbnetc_100            | 128 |   nan    |
|            fbnetv3_b            | 128 |   nan    |
|            gernet_l             | 128 |   nan    |
|          ghostnet_100           | 128 |   nan    |
|       gluon_inception_v3        | 128 |   nan    |
|        gluon_xception65         | 32  |   nan    |
|          gmixer_24_224          | 64  |   nan    |
|          gmlp_s16_224           | 64  |   nan    |
|            hrnet_w18            |  2  |   nan    |
|          inception_v3           | 128 |   nan    |
|          jx_nest_base           | 32  |   nan    |
|            lcnet_050            | 128 |   nan    |
|          mixer_b16_224          | 64  |   nan    |
|            mixnet_l             | 64  |   nan    |
|           mnasnet_100           | 128 |   nan    |
|         mobilenetv2_100         | 128 |   nan    |
|      mobilenetv3_large_100      | 128 |   nan    |
|           mobilevit_s           | 32  |   nan    |
|            nfnet_l0             | 64  |   nan    |
|            pit_b_224            | 64  |   nan    |
|          pnasnet5large          | 16  |   nan    |
|         poolformer_m36          | 64  |   nan    |
|           regnety_002           | 128 |   nan    |
|            repvgg_a2            | 128 |   nan    |
|        res2net101_26w_4s        | 64  |   nan    |
|        res2net50_14w_8s         |  2  |   nan    |
|           res2next50            |  2  |   nan    |
|          resmlp_12_224          | 128 |   nan    |
|           resnest101e           | 32  |   nan    |
|           rexnet_100            | 128 |   nan    |
|        sebotnet33ts_256         |  0  |   nan    |
|           selecsls42b           | 128 |   nan    |
|          spnasnet_100           | 128 |   nan    |
|  swin_base_patch4_window7_224   | 64  |   nan    |
|     swsl_resnext101_32x16d      | 32  |   nan    |
|       tf_efficientnet_b0        | 128 |   nan    |
|           tf_mixnet_l           | 64  |   nan    |
|            tinynet_a            | 128 |   nan    |
|        tnt_s_patch16_224        | 64  |   nan    |
|        twins_pcpvt_base         | 32  |   nan    |
|         visformer_small         | 128 |   nan    |
|      vit_base_patch16_224       | 64  |   nan    |
|           volo_d1_224           | 64  |   nan    |
|      xcit_large_24_p8_224       |  5  |   nan    |
+---------------------------------+-----+----------+

[jansel]

This is inference? Or training?

[blzheng]

Inference.

[blzheng]

Performance Dashboard for float32 precision -- Single-Socket Multi-threads

Executive Summary

see more

We evaluate different backends across three benchmark suites - torchbench, huggingface and timm. We run these experiments on Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz. Each experiment runs one iteration of forward pass. For accuracy, we check the numerical correctness of forward pass outputs by comparing with native pytorch. We measure speedup by normalizing against the performance of native pytorch. We report mean compilation latency numbers and peak memory footprint reduction ratio.

Caveats

1. Batch size has been reduced to workaround OOM errors. Work is in progress to reduce peak memory footprint.
2. Experiments do not cover dynamic shapes.
3. Experimental setup does not have optimizer.

To measure performance, compilation latency and memory footprint reduction, we remove the models that fail accuracy checks.

Passrate

+----------+------------+-------------+-------------+
| Compiler | torchbench | huggingface | timm_models |
+----------+------------+-------------+-------------+
| inductor | 93%, 51/55 | 100%, 44/44 | 90%, 55/61  |
+----------+------------+-------------+-------------+

Geometric mean speedup

+----------+------------+-------------+-------------+
| Compiler | torchbench | huggingface | timm_models |
+----------+------------+-------------+-------------+
| inductor |   1.03x    |    1.12x    |    1.03x    |
+----------+------------+-------------+-------------+

Mean compilation time (seconds)

+----------+------------+-------------+-------------+
| Compiler | torchbench | huggingface | timm_models |
+----------+------------+-------------+-------------+
| inductor |    3.70    |    5.17     |    5.74     |
+----------+------------+-------------+-------------+

Peak memory footprint compression ratio (higher is better)

+----------+------------+-------------+-------------+
| Compiler | torchbench | huggingface | timm_models |
+----------+------------+-------------+-------------+
| inductor |   0.94x    |    0.98x    |    0.94x    |
+----------+------------+-------------+-------------+

torchbench suite with float32 precision

see more

Performance speedup

+-----------------------------------+------+----------+
|               name                |  bs  | inductor |
+-----------------------------------+------+----------+
|           timm_resnest            |  32  |  1.2382  |
|        shufflenet_v2_x1_0         |  64  |  1.2261  |
|            densenet121            |  64  |  1.134   |
|   pytorch_CycleGAN_and_pix2pix    |  1   |  1.1287  |
|           squeezenet1_1           |  16  |  1.1201  |
|             resnet50              |  32  |  1.1148  |
|         soft_actor_critic         | 256  |  1.1113  |
|              alexnet              | 128  |  1.1105  |
|            mnasnet1_0             |  32  |  1.0631  |
|               dlrm                | 2048 |  1.046   |
|                drq                |  1   |  1.0436  |
|             resnet18              |  8   |  1.036   |
|            Super_SloMo            |  6   |  1.0339  |
|            timm_regnet            |  32  |  1.0339  |
|            hf_T5_large            |  1   |  1.027   |
|            hf_BigBird             |  1   |  1.0227  |
|            hf_Reformer            |  1   |  1.0173  |
|               vgg16               |  4   |  1.0146  |
|          pytorch_stargan          |  16  |  1.013   |
|      resnet50_quantized_qat       |  32  |  1.0099  |
|    mobilenet_v2_quantized_qat     |  96  |  1.0007  |
|              demucs               |  1   |  0.9942  |
|            tts_angular            |  64  |  0.9912  |
|               hf_T5               |  1   |  0.9896  |
|       functorch_dp_cifar10        |  64  |  0.9846  |
|               dcgan               | 256  |  0.9822  |
|            timm_vovnet            |  32  |  0.9784  |
|           BERT_pytorch            |  2   |  0.9668  |
|          LearningToPaint          |  96  |  0.9624  |
|        Background_Matting         |  1   |  0.9616  |
|        mobilenet_v3_large         |  32  |  0.9279  |
|           pytorch_unet            |  1   |  0.9242  |
|           hf_Longformer           |  1   |  0.9203  |
|              hf_GPT2              |  1   |  0.9074  |
|           hf_GPT2_large           |  1   |  0.8962  |
|        speech_transformer         |  1   |  0.8892  |
|             hf_Albert             |  1   |  0.8793  |
|   timm_vision_transformer_large   |  8   |  0.8567  |
|            hf_T5_base             |  1   |  0.837   |
|           hf_DistilBert           |  1   |  0.8303  |
|      nvidia_deeprecommender       | 256  |  0.8219  |
|              hf_Bert              |  1   |  0.7961  |
|              yolov3               |  8   |  0.784   |
|              hf_Bart              |  1   |  0.7805  |
|           fastNLP_Bert            |  1   |  0.7779  |
|           mobilenet_v2            |  16  |  0.7575  |
|            timm_nfnet             | 128  |  0.7403  |
|          vision_maskrcnn          |  1   |  0.739   |
|          resnext50_32x4d          |  8   |  0.7271  |
|      timm_vision_transformer      |  8   |  0.7141  |
| attention_is_all_you_need_pytorch |  32  |  0.704   |
|         timm_efficientnet         |  64  |  0.6645  |
|           lennard_jones           | 1000 |  0.3621  |
|             tacotron2             |  0   |   0.0    |
|     detectron2_fcos_r_50_fpn      |  0   |   0.0    |
+-----------------------------------+------+----------+

Accuracy

+-----------------------------------+-----+------------------+
|               name                | bs  |     inductor     |
+-----------------------------------+-----+------------------+
|            hf_T5_large            |  2  | pass_due_to_skip |
|   timm_vision_transformer_large   |  2  | pass_due_to_skip |
|           hf_GPT2_large           |  2  | pass_due_to_skip |
|           BERT_pytorch            |  2  |       pass       |
|        shufflenet_v2_x1_0         |  2  |       pass       |
|        mobilenet_v3_large         |  2  |       pass       |
|      nvidia_deeprecommender       |  2  |       pass       |
|   pytorch_CycleGAN_and_pix2pix    |  1  |       pass       |
|          pytorch_stargan          | 16  |       pass       |
|           pytorch_unet            |  2  |       pass       |
|             resnet18              |  2  |       pass       |
|             resnet50              |  2  |       pass       |
|      resnet50_quantized_qat       |  2  |       pass       |
|          resnext50_32x4d          |  2  |       pass       |
|         soft_actor_critic         | 256 |       pass       |
|           mobilenet_v2            |  2  |       pass       |
|        speech_transformer         |  2  |       pass       |
|           squeezenet1_1           |  2  |       pass       |
|         timm_efficientnet         |  2  |       pass       |
|            timm_nfnet             |  2  |       pass       |
|            timm_regnet            |  2  |       pass       |
|           timm_resnest            |  2  |       pass       |
|      timm_vision_transformer      |  2  |       pass       |
|            timm_vovnet            |  2  |       pass       |
|            tts_angular            |  2  |       pass       |
|               vgg16               |  2  |       pass       |
|    mobilenet_v2_quantized_qat     |  2  |       pass       |
|            mnasnet1_0             |  2  |       pass       |
|           fastNLP_Bert            |  2  |       pass       |
|       functorch_dp_cifar10        |  2  |       pass       |
|          LearningToPaint          |  2  |       pass       |
|            Super_SloMo            |  2  |       pass       |
|              alexnet              |  2  |       pass       |
| attention_is_all_you_need_pytorch |  2  |       pass       |
|               dcgan               |  2  |       pass       |
|              demucs               |  1  |       pass       |
|            densenet121            |  2  |       pass       |
|               dlrm                |  2  |       pass       |
|                drq                |  1  |       pass       |
|           lennard_jones           |  2  |       pass       |
|              yolov3               |  2  |       pass       |
|             hf_Albert             |  2  |       pass       |
|              hf_Bart              |  2  |       pass       |
|              hf_Bert              |  2  |       pass       |
|            hf_BigBird             |  2  |       pass       |
|           hf_DistilBert           |  2  |       pass       |
|              hf_GPT2              |  2  |       pass       |
|           hf_Longformer           |  2  |       pass       |
|            hf_Reformer            |  2  |       pass       |
|               hf_T5               |  2  |       pass       |
|            hf_T5_base             |  2  |       pass       |
|        Background_Matting         |  1  |  fail_accuracy   |
|             tacotron2             |  0  |      0.0000      |
|     detectron2_fcos_r_50_fpn      |  0  |      0.0000      |
|          vision_maskrcnn          |  0  |      0.0000      |
+-----------------------------------+-----+------------------+

Compilation latency (sec)

+-----------------------------------+------+----------+
|               name                |  bs  | inductor |
+-----------------------------------+------+----------+
|            hf_T5_large            |  1   | 19.5685  |
|          vision_maskrcnn          |  1   | 14.1261  |
|           hf_Longformer           |  1   |  10.801  |
|           hf_GPT2_large           |  1   | 10.5252  |
|   timm_vision_transformer_large   |  8   | 10.0167  |
|            hf_T5_base             |  1   |  9.5113  |
|            hf_BigBird             |  1   |  9.4953  |
|              yolov3               |  8   |  9.4608  |
|            timm_nfnet             | 128  |  8.7221  |
|            densenet121            |  64  |  8.1788  |
|            Super_SloMo            |  6   |  5.0844  |
|        speech_transformer         |  1   |  4.3803  |
|              hf_Bart              |  1   |  4.2628  |
|            timm_regnet            |  32  |  3.9241  |
| attention_is_all_you_need_pytorch |  32  |  3.9061  |
|               hf_T5               |  1   |  3.8853  |
|           fastNLP_Bert            |  1   |  3.8334  |
|         timm_efficientnet         |  64  |  3.6686  |
|           BERT_pytorch            |  2   |  3.6197  |
|              hf_Bert              |  1   |  3.3367  |
|              hf_GPT2              |  1   |  3.064   |
|        shufflenet_v2_x1_0         |  64  |  2.9784  |
|        Background_Matting         |  1   |  2.8748  |
|      timm_vision_transformer      |  8   |  2.8561  |
|        mobilenet_v3_large         |  32  |  2.7998  |
|           mobilenet_v2            |  16  |  2.7577  |
|            hf_Reformer            |  1   |  2.7114  |
|             hf_Albert             |  1   |  2.6651  |
|          resnext50_32x4d          |  8   |  2.6175  |
|            timm_vovnet            |  32  |  2.521   |
|            mnasnet1_0             |  32  |  2.4095  |
|             resnet50              |  32  |  2.3808  |
|           timm_resnest            |  32  |  1.8313  |
|           hf_DistilBert           |  1   |  1.7493  |
|       functorch_dp_cifar10        |  64  |  1.5456  |
|           pytorch_unet            |  1   |  1.4478  |
|   pytorch_CycleGAN_and_pix2pix    |  1   |  1.3776  |
|          pytorch_stargan          |  16  |  1.1897  |
|          LearningToPaint          |  96  |  1.0779  |
|             resnet18              |  8   |  0.9023  |
|           squeezenet1_1           |  16  |  0.7238  |
|              demucs               |  1   |  0.7217  |
|               dlrm                | 2048 |  0.6254  |
|               vgg16               |  4   |  0.4764  |
|            tts_angular            |  64  |  0.4255  |
|              alexnet              | 128  |  0.3327  |
|                drq                |  1   |  0.3197  |
|      nvidia_deeprecommender       | 256  |  0.3053  |
|         soft_actor_critic         | 256  |  0.1629  |
|           lennard_jones           | 1000 |  0.1179  |
|               dcgan               | 256  |  0.1104  |
|    mobilenet_v2_quantized_qat     |  96  | -0.0207  |
|      resnet50_quantized_qat       |  32  | -0.1267  |
|     detectron2_fcos_r_50_fpn      |  0   |   nan    |
|             tacotron2             |  0   |   nan    |
+-----------------------------------+------+----------+

Peak Memory Compression Ratio

+-----------------------------------+------+----------+
|               name                |  bs  | inductor |
+-----------------------------------+------+----------+
|              demucs               |  1   |  0.9985  |
|      nvidia_deeprecommender       | 256  |  0.9969  |
|    mobilenet_v2_quantized_qat     |  96  |  0.9967  |
|            hf_T5_base             |  1   |  0.9947  |
|               vgg16               |  4   |  0.994   |
|           hf_GPT2_large           |  1   |  0.9939  |
|        Background_Matting         |  1   |  0.9934  |
|           pytorch_unet            |  1   |  0.9917  |
|            timm_nfnet             | 128  |  0.9916  |
|          LearningToPaint          |  96  |  0.9915  |
|              alexnet              | 128  |  0.991   |
|            hf_BigBird             |  1   |  0.9909  |
|            Super_SloMo            |  6   |  0.9886  |
|           lennard_jones           | 1000 |  0.9865  |
|         soft_actor_critic         | 256  |  0.9834  |
| attention_is_all_you_need_pytorch |  32  |  0.9832  |
|           hf_DistilBert           |  1   |  0.982   |
|            tts_angular            |  64  |  0.9816  |
|              hf_GPT2              |  1   |  0.9809  |
|           BERT_pytorch            |  2   |  0.9804  |
|                drq                |  1   |  0.9799  |
|              hf_Bart              |  1   |  0.9795  |
|           fastNLP_Bert            |  1   |  0.9793  |
|         timm_efficientnet         |  64  |  0.9791  |
|           mobilenet_v2            |  16  |  0.9776  |
|        speech_transformer         |  1   |  0.9734  |
|              hf_Bert              |  1   |  0.9733  |
|      resnet50_quantized_qat       |  32  |  0.972   |
|            timm_vovnet            |  32  |  0.972   |
|               hf_T5               |  1   |  0.9719  |
|   timm_vision_transformer_large   |  8   |  0.9702  |
|           hf_Longformer           |  1   |  0.9675  |
|        mobilenet_v3_large         |  32  |  0.9621  |
|          vision_maskrcnn          |  1   |  0.9553  |
|               dlrm                | 2048 |  0.9471  |
|             hf_Albert             |  1   |  0.9471  |
|             resnet50              |  32  |  0.9328  |
|           squeezenet1_1           |  16  |  0.931   |
|       functorch_dp_cifar10        |  64  |  0.9216  |
|      timm_vision_transformer      |  8   |  0.9087  |
|            hf_T5_large            |  1   |  0.9059  |
|   pytorch_CycleGAN_and_pix2pix    |  1   |  0.8997  |
|            mnasnet1_0             |  32  |  0.8658  |
|          resnext50_32x4d          |  8   |  0.8647  |
|        shufflenet_v2_x1_0         |  64  |  0.8498  |
|             resnet18              |  8   |  0.8451  |
|          pytorch_stargan          |  16  |  0.845   |
|            hf_Reformer            |  1   |  0.812   |
|              yolov3               |  8   |  0.7951  |
|            densenet121            |  64  |  0.7917  |
|           timm_resnest            |  32  |  0.7725  |
|            timm_regnet            |  32  |  0.7187  |
|               dcgan               | 256  |  0.6787  |
|     detectron2_fcos_r_50_fpn      |  0   |   nan    |
|             tacotron2             |  0   |   nan    |
+-----------------------------------+------+----------+

huggingface suite with float32 precision

see more

Performance speedup

+-----------------------------------------+----+----------+
|                  name                   | bs | inductor |
+-----------------------------------------+----+----------+
|            XLNetLMHeadModel             | 4  |  7.3631  |
|       MT5ForConditionalGeneration       | 2  |  1.8516  |
|             XGLMForCausalLM             | 1  |  1.3518  |
|     M2M100ForConditionalGeneration      | 2  |  1.3156  |
|               DistillGPT2               | 1  |  1.2913  |
|           ElectraForCausalLM            | 1  |  1.283   |
|          MobileBertForMaskedLM          | 16 |  1.2617  |
|               GoogleFnet                | 1  |  1.2474  |
|             OPTForCausalLM              | 4  |  1.2442  |
|            YituTechConvBert             | 1  |  1.1842  |
|     MobileBertForQuestionAnswering      | 32 |  1.1796  |
|                 BigBird                 | 1  |  1.0738  |
|          AllenaiLongformerBase          | 1  |  1.0663  |
|            AlbertForMaskedLM            | 2  |  1.0232  |
|       AlbertForQuestionAnswering        | 2  |  1.0177  |
|     PegasusForConditionalGeneration     | 4  |  0.9945  |
|         MegatronBertForCausalLM         | 2  |  0.9931  |
|                 T5Small                 | 1  |  0.9825  |
|           RobertaForCausalLM            | 4  |  0.9662  |
|           PegasusForCausalLM            | 8  |  0.9613  |
|            TrOCRForCausalLM             | 8  |  0.9557  |
|           DebertaForMaskedLM            | 4  |  0.9541  |
|     PLBartForConditionalGeneration      | 8  |  0.9521  |
|                CamemBert                | 1  |  0.947   |
|       DebertaForQuestionAnswering       | 4  |  0.9268  |
|            PLBartForCausalLM            | 16 |  0.9236  |
|      MBartForConditionalGeneration      | 8  |  0.9134  |
|      GPT2ForSequenceClassification      | 4  |  0.9105  |
|         Speech2Text2ForCausalLM         | 64 |  0.9003  |
|          DistilBertForMaskedLM          | 16 |  0.8957  |
|            MBartForCausalLM             | 16 |  0.8906  |
|    MegatronBertForQuestionAnswering     | 8  |  0.8662  |
|       RobertaForQuestionAnswering       | 64 |  0.8525  |
|    LayoutLMForSequenceClassification    | 16 |  0.8496  |
|           LayoutLMForMaskedLM           | 16 |  0.846   |
|       T5ForConditionalGeneration        | 4  |  0.8412  |
|             BartForCausalLM             | 2  |  0.828   |
|             BertForMaskedLM             | 64 |  0.8275  |
|        BertForQuestionAnswering         | 64 |  0.8246  |
|     DistilBertForQuestionAnswering      | 32 |  0.8142  |
|       ElectraForQuestionAnswering       | 64 |  0.7925  |
| BlenderbotSmallForConditionalGeneration | 32 |  0.788   |
|      BartForConditionalGeneration       | 1  |  0.7823  |
|       BlenderbotSmallForCausalLM        | 64 |  0.7666  |
+-----------------------------------------+----+----------+

Accuracy

+-----------------------------------------+----+----------+
|                  name                   | bs | inductor |
+-----------------------------------------+----+----------+
|            AlbertForMaskedLM            | 1  |   pass   |
|       AlbertForQuestionAnswering        | 1  |   pass   |
|      MBartForConditionalGeneration      | 1  |   pass   |
|       MT5ForConditionalGeneration       | 1  |   pass   |
|         MegatronBertForCausalLM         | 1  |   pass   |
|    MegatronBertForQuestionAnswering     | 1  |   pass   |
|          MobileBertForMaskedLM          | 1  |   pass   |
|     MobileBertForQuestionAnswering      | 1  |   pass   |
|             OPTForCausalLM              | 1  |   pass   |
|            PLBartForCausalLM            | 1  |   pass   |
|     PLBartForConditionalGeneration      | 1  |   pass   |
|           PegasusForCausalLM            | 1  |   pass   |
|     PegasusForConditionalGeneration     | 1  |   pass   |
|           RobertaForCausalLM            | 1  |   pass   |
|       RobertaForQuestionAnswering       | 1  |   pass   |
|         Speech2Text2ForCausalLM         | 1  |   pass   |
|       T5ForConditionalGeneration        | 1  |   pass   |
|                 T5Small                 | 1  |   pass   |
|            TrOCRForCausalLM             | 1  |   pass   |
|             XGLMForCausalLM             | 1  |   pass   |
|            XLNetLMHeadModel             | 1  |   pass   |
|            MBartForCausalLM             | 1  |   pass   |
|     M2M100ForConditionalGeneration      | 1  |   pass   |
|    LayoutLMForSequenceClassification    | 1  |   pass   |
|                CamemBert                | 1  |   pass   |
|          AllenaiLongformerBase          | 1  |   pass   |
|             BartForCausalLM             | 1  |   pass   |
|      BartForConditionalGeneration       | 1  |   pass   |
|             BertForMaskedLM             | 1  |   pass   |
|        BertForQuestionAnswering         | 1  |   pass   |
|                 BigBird                 | 1  |   pass   |
|       BlenderbotSmallForCausalLM        | 1  |   pass   |
| BlenderbotSmallForConditionalGeneration | 1  |   pass   |
|           DebertaForMaskedLM            | 1  |   pass   |
|           LayoutLMForMaskedLM           | 1  |   pass   |
|       DebertaForQuestionAnswering       | 1  |   pass   |
|          DistilBertForMaskedLM          | 1  |   pass   |
|     DistilBertForQuestionAnswering      | 1  |   pass   |
|               DistillGPT2               | 1  |   pass   |
|           ElectraForCausalLM            | 1  |   pass   |
|       ElectraForQuestionAnswering       | 1  |   pass   |
|      GPT2ForSequenceClassification      | 1  |   pass   |
|               GoogleFnet                | 1  |   pass   |
|            YituTechConvBert             | 1  |   pass   |
+-----------------------------------------+----+----------+

Compilation latency (sec)

+-----------------------------------------+----+----------+
|                  name                   | bs | inductor |
+-----------------------------------------+----+----------+
|          MobileBertForMaskedLM          | 16 | 12.2183  |
|     MobileBertForQuestionAnswering      | 32 | 11.9702  |
|          AllenaiLongformerBase          | 1  | 11.5861  |
|                 BigBird                 | 1  |  9.5059  |
|      BartForConditionalGeneration       | 1  |  8.9487  |
|      MBartForConditionalGeneration      | 8  |  8.8587  |
|     PegasusForConditionalGeneration     | 4  |  8.4913  |
|           DebertaForMaskedLM            | 4  |  8.4733  |
|       DebertaForQuestionAnswering       | 4  |  8.4294  |
|     M2M100ForConditionalGeneration      | 2  |  7.602   |
|    MegatronBertForQuestionAnswering     | 8  |  7.4598  |
|         MegatronBertForCausalLM         | 2  |  6.4697  |
|             XGLMForCausalLM             | 1  |  6.4641  |
| BlenderbotSmallForConditionalGeneration | 32 |   5.96   |
|            YituTechConvBert             | 1  |  5.2836  |
|       T5ForConditionalGeneration        | 4  |  4.8153  |
|           LayoutLMForMaskedLM           | 16 |  4.5577  |
|       ElectraForQuestionAnswering       | 64 |  4.5315  |
|     PLBartForConditionalGeneration      | 8  |  4.4101  |
|             BertForMaskedLM             | 64 |  4.3663  |
|    LayoutLMForSequenceClassification    | 16 |  4.2891  |
|        BertForQuestionAnswering         | 64 |  4.1585  |
|       RobertaForQuestionAnswering       | 64 |  4.1072  |
|                 T5Small                 | 1  |  3.8782  |
|       MT5ForConditionalGeneration       | 2  |  3.8328  |
|           RobertaForCausalLM            | 4  |  3.8004  |
|             BartForCausalLM             | 2  |  3.7592  |
|      GPT2ForSequenceClassification      | 4  |  3.7104  |
|            MBartForCausalLM             | 16 |  3.6174  |
|           PegasusForCausalLM            | 8  |  3.5072  |
|                CamemBert                | 1  |  3.3432  |
|           ElectraForCausalLM            | 1  |  3.3289  |
|             OPTForCausalLM              | 4  |  3.3105  |
|            TrOCRForCausalLM             | 8  |  3.234   |
|       AlbertForQuestionAnswering        | 2  |  2.935   |
|            AlbertForMaskedLM            | 2  |  2.8877  |
|       BlenderbotSmallForCausalLM        | 64 |  2.7639  |
|               GoogleFnet                | 1  |  2.0909  |
|          DistilBertForMaskedLM          | 16 |  1.9801  |
|     DistilBertForQuestionAnswering      | 32 |  1.9167  |
|            PLBartForCausalLM            | 16 |  1.9065  |
|         Speech2Text2ForCausalLM         | 64 |  1.8383  |
|               DistillGPT2               | 1  |  1.6408  |
|            XLNetLMHeadModel             | 4  | -16.2899 |
+-----------------------------------------+----+----------+

Peak Memory Compression Ratio

+-----------------------------------------+----+----------+
|                  name                   | bs | inductor |
+-----------------------------------------+----+----------+
|            AlbertForMaskedLM            | 2  |  0.9966  |
|       ElectraForQuestionAnswering       | 64 |  0.9966  |
|       AlbertForQuestionAnswering        | 2  |  0.9964  |
|           LayoutLMForMaskedLM           | 16 |  0.9961  |
|       BlenderbotSmallForCausalLM        | 64 |  0.996   |
|    LayoutLMForSequenceClassification    | 16 |  0.9956  |
|      GPT2ForSequenceClassification      | 4  |  0.9956  |
|             BertForMaskedLM             | 64 |  0.9955  |
|        BertForQuestionAnswering         | 64 |  0.9942  |
|       RobertaForQuestionAnswering       | 64 |  0.9942  |
|           DebertaForMaskedLM            | 4  |  0.9939  |
|       T5ForConditionalGeneration        | 4  |  0.9938  |
|             BartForCausalLM             | 2  |  0.9934  |
|       DebertaForQuestionAnswering       | 4  |  0.9927  |
|                 BigBird                 | 1  |  0.9924  |
|            XLNetLMHeadModel             | 4  |  0.9916  |
|            MBartForCausalLM             | 16 |  0.9915  |
|            PLBartForCausalLM            | 16 |  0.9902  |
| BlenderbotSmallForConditionalGeneration | 32 |  0.9902  |
|      BartForConditionalGeneration       | 1  |  0.9901  |
|           PegasusForCausalLM            | 8  |  0.9896  |
|     PegasusForConditionalGeneration     | 4  |  0.9887  |
|               GoogleFnet                | 1  |  0.9885  |
|         Speech2Text2ForCausalLM         | 64 |  0.988   |
|       MT5ForConditionalGeneration       | 2  |  0.9876  |
|          DistilBertForMaskedLM          | 16 |  0.9871  |
|      MBartForConditionalGeneration      | 8  |  0.9865  |
|            TrOCRForCausalLM             | 8  |  0.986   |
|     M2M100ForConditionalGeneration      | 2  |  0.9859  |
|               DistillGPT2               | 1  |  0.9851  |
|    MegatronBertForQuestionAnswering     | 8  |  0.985   |
|             XGLMForCausalLM             | 1  |  0.9834  |
|             OPTForCausalLM              | 4  |  0.9829  |
|     PLBartForConditionalGeneration      | 8  |  0.9812  |
|          AllenaiLongformerBase          | 1  |  0.9789  |
|                CamemBert                | 1  |  0.9726  |
|                 T5Small                 | 1  |  0.9719  |
|           RobertaForCausalLM            | 4  |  0.9716  |
|            YituTechConvBert             | 1  |  0.9622  |
|         MegatronBertForCausalLM         | 2  |  0.9588  |
|           ElectraForCausalLM            | 1  |  0.944   |
|          MobileBertForMaskedLM          | 16 |  0.9269  |
|     DistilBertForQuestionAnswering      | 32 |  0.9081  |
|     MobileBertForQuestionAnswering      | 32 |  0.8899  |
+-----------------------------------------+----+----------+

timm_models suite with float32 precision

see more

Performance speedup

+---------------------------------+-----+----------+
|              name               | bs  | inductor |
+---------------------------------+-----+----------+
|          inception_v3           | 128 |  1.2067  |
|        res2net101_26w_4s        | 64  |  1.189   |
|        adv_inception_v3         | 128 |  1.1879  |
|       gluon_inception_v3        | 128 |  1.1847  |
|           mnasnet_100           | 128 |  1.1736  |
|          spnasnet_100           | 128 |  1.1705  |
|        convmixer_768_32         | 32  |  1.1569  |
|          ghostnet_100           | 128 |  1.1505  |
|           fbnetc_100            | 128 |  1.134   |
|           resnest101e           | 32  |  1.1296  |
|             dpn107              | 32  |  1.1117  |
|           volo_d1_224           | 64  |  1.1116  |
|            gernet_l             | 128 |  1.1023  |
|          pnasnet5large          | 16  |  1.0403  |
|           selecsls42b           | 128 |  1.0238  |
|      mobilenetv3_large_100      | 128 |  1.0237  |
|        res2net50_14w_8s         |  2  |  1.0053  |
|            repvgg_a2            | 128 |  1.0013  |
|          gmixer_24_224          | 64  |  0.9843  |
|         poolformer_m36          | 64  |  0.9821  |
|            hrnet_w18            |  2  |  0.9727  |
|         mobilenetv2_100         | 128 |  0.9628  |
|             dla102              | 64  |  0.951   |
|      xcit_large_24_p8_224       |  5  |  0.9447  |
|            fbnetv3_b            | 128 |   0.93   |
|           tf_mixnet_l           | 64  |  0.9279  |
|          cait_m36_384           |  2  |  0.9264  |
|        ese_vovnet19b_dw         | 128 |  0.9251  |
|           regnety_002           | 128 |  0.9222  |
|            nfnet_l0             | 64  |  0.9063  |
|            mixnet_l             | 64  |  0.8793  |
|     swsl_resnext101_32x16d      | 32  |  0.8755  |
|      beit_base_patch16_224      | 64  |  0.8658  |
|            lcnet_050            | 128 |  0.8585  |
|  swin_base_patch4_window7_224   | 64  |  0.8467  |
| deit_base_distilled_patch16_224 | 64  |  0.8454  |
|      vit_base_patch16_224       | 64  |  0.8451  |
|           convit_base           | 32  |  0.8401  |
|           dm_nfnet_f0           | 128 |  0.8236  |
|        tnt_s_patch16_224        | 64  |  0.8202  |
|          resmlp_12_224          | 128 |  0.8106  |
|          mixer_b16_224          | 64  |  0.809   |
|          jx_nest_base           | 32  |  0.8065  |
|           rexnet_100            | 128 |   0.8    |
|         visformer_small         | 128 |  0.7974  |
|          convnext_base          | 32  |  0.7851  |
|            pit_b_224            | 64  |  0.7825  |
|           res2next50            |  2  |  0.7694  |
|         coat_lite_mini          | 128 |  0.736   |
|        twins_pcpvt_base         | 32  |  0.7297  |
|            tinynet_a            | 128 |  0.7163  |
|       tf_efficientnet_b0        | 128 |  0.7025  |
|        gluon_xception65         | 32  |  0.6939  |
|         crossvit_9_240          | 64  |  0.6874  |
|          cspdarknet53           | 64  |  0.6632  |
|          gmlp_s16_224           | 64  |  0.553   |
|           mobilevit_s           | 32  |  0.4792  |
|        sebotnet33ts_256         |  0  |   0.0    |
|       eca_botnext26ts_256       |  0  |   0.0    |
|        eca_halonext26ts         |  0  |   0.0    |
|          botnet26t_256          |  0  |   0.0    |
+---------------------------------+-----+----------+

Accuracy

+---------------------------------+----+---------------+
|              name               | bs |   inductor    |
+---------------------------------+----+---------------+
|        adv_inception_v3         | 2  |     pass      |
|      beit_base_patch16_224      | 2  |     pass      |
|         mobilenetv2_100         | 2  |     pass      |
|      mobilenetv3_large_100      | 2  |     pass      |
|           mobilevit_s           | 2  |     pass      |
|            nfnet_l0             | 2  |     pass      |
|            pit_b_224            | 2  |     pass      |
|          pnasnet5large          | 2  |     pass      |
|         poolformer_m36          | 2  |     pass      |
|           regnety_002           | 2  |     pass      |
|            repvgg_a2            | 2  |     pass      |
|        res2net101_26w_4s        | 2  |     pass      |
|        res2net50_14w_8s         | 2  |     pass      |
|           res2next50            | 2  |     pass      |
|          resmlp_12_224          | 2  |     pass      |
|           resnest101e           | 2  |     pass      |
|           rexnet_100            | 2  |     pass      |
|           selecsls42b           | 2  |     pass      |
|          spnasnet_100           | 2  |     pass      |
|  swin_base_patch4_window7_224   | 2  |     pass      |
|     swsl_resnext101_32x16d      | 2  |     pass      |
|       tf_efficientnet_b0        | 2  |     pass      |
|           tf_mixnet_l           | 2  |     pass      |
|            tinynet_a            | 2  |     pass      |
|        tnt_s_patch16_224        | 2  |     pass      |
|        twins_pcpvt_base         | 2  |     pass      |
|         visformer_small         | 2  |     pass      |
|      vit_base_patch16_224       | 2  |     pass      |
|           volo_d1_224           | 2  |     pass      |
|           mnasnet_100           | 2  |     pass      |
|            mixnet_l             | 2  |     pass      |
|          mixer_b16_224          | 2  |     pass      |
|        ese_vovnet19b_dw         | 2  |     pass      |
|         coat_lite_mini          | 2  |     pass      |
|           convit_base           | 2  |     pass      |
|        convmixer_768_32         | 2  |     pass      |
|          convnext_base          | 2  |     pass      |
|         crossvit_9_240          | 2  |     pass      |
|          cspdarknet53           | 2  |     pass      |
| deit_base_distilled_patch16_224 | 2  |     pass      |
|             dla102              | 2  |     pass      |
|           dm_nfnet_f0           | 2  |     pass      |
|             dpn107              | 2  |     pass      |
|            lcnet_050            | 2  |     pass      |
|      xcit_large_24_p8_224       | 2  |     pass      |
|           fbnetc_100            | 2  |     pass      |
|            fbnetv3_b            | 2  |     pass      |
|            gernet_l             | 2  |     pass      |
|       gluon_inception_v3        | 2  |     pass      |
|        gluon_xception65         | 2  |     pass      |
|          gmixer_24_224          | 2  |     pass      |
|          gmlp_s16_224           | 2  |     pass      |
|            hrnet_w18            | 2  |     pass      |
|          inception_v3           | 2  |     pass      |
|          jx_nest_base           | 2  |     pass      |
|        eca_halonext26ts         | 2  |  fail_to_run  |
|        sebotnet33ts_256         | 2  |  fail_to_run  |
|          botnet26t_256          | 2  |  fail_to_run  |
|       eca_botnext26ts_256       | 2  |  fail_to_run  |
|          ghostnet_100           | 2  | fail_accuracy |
|          cait_m36_384           | 2  | fail_accuracy |
+---------------------------------+----+---------------+

Compilation latency (sec)

+---------------------------------+-----+----------+
|              name               | bs  | inductor |
+---------------------------------+-----+----------+
|            hrnet_w18            |  2  | 17.4435  |
|          pnasnet5large          | 16  | 15.8176  |
|           mobilevit_s           | 32  | 11.8116  |
|      xcit_large_24_p8_224       |  5  | 11.6298  |
|  swin_base_patch4_window7_224   | 64  | 11.2435  |
|          cait_m36_384           |  2  |  11.183  |
|         poolformer_m36          | 64  |  9.7591  |
|           dm_nfnet_f0           | 128 |  9.0595  |
|        twins_pcpvt_base         | 32  |  9.0463  |
|          jx_nest_base           | 32  |  8.8532  |
|        res2net101_26w_4s        | 64  |  8.7024  |
|           tf_mixnet_l           | 64  |  8.1885  |
|        res2net50_14w_8s         |  2  |  8.0084  |
|        gluon_xception65         | 32  |  7.7414  |
|            mixnet_l             | 64  |  7.7377  |
|        tnt_s_patch16_224        | 64  |  7.3083  |
|             dpn107              | 32  |  7.2266  |
|          gmlp_s16_224           | 64  |  7.1444  |
|            fbnetv3_b            | 128 |  6.8568  |
|           volo_d1_224           | 64  |  6.0621  |
|          cspdarknet53           | 64  |  5.9534  |
|             dla102              | 64  |  5.8048  |
|          convnext_base          | 32  |  5.558   |
|         crossvit_9_240          | 64  |  5.5577  |
|           rexnet_100            | 128 |  5.1342  |
|            nfnet_l0             | 64  |  5.0545  |
|       tf_efficientnet_b0        | 128 |  4.8415  |
|     swsl_resnext101_32x16d      | 32  |  4.7578  |
|            pit_b_224            | 64  |  4.7423  |
|           res2next50            |  2  |  4.6471  |
|          inception_v3           | 128 |  4.617   |
|            tinynet_a            | 128 |  4.4623  |
|          ghostnet_100           | 128 |  4.3717  |
|        adv_inception_v3         | 128 |  4.248   |
|       gluon_inception_v3        | 128 |  4.2008  |
|      beit_base_patch16_224      | 64  |  4.1799  |
|         coat_lite_mini          | 128 |  4.0925  |
|           convit_base           | 32  |  4.0284  |
|           fbnetc_100            | 128 |  3.6995  |
| deit_base_distilled_patch16_224 | 64  |  3.6688  |
|          gmixer_24_224          | 64  |  3.5157  |
|          spnasnet_100           | 128 |  3.5142  |
|         visformer_small         | 128 |  3.4201  |
|          mixer_b16_224          | 64  |  3.2746  |
|            repvgg_a2            | 128 |  3.2501  |
|      vit_base_patch16_224       | 64  |  3.0714  |
|           regnety_002           | 128 |  3.0702  |
|            gernet_l             | 128 |  3.0452  |
|      mobilenetv3_large_100      | 128 |  3.0294  |
|           mnasnet_100           | 128 |  2.8618  |
|         mobilenetv2_100         | 128 |  2.7368  |
|        ese_vovnet19b_dw         | 128 |  2.3969  |
|          resmlp_12_224          | 128 |  2.1207  |
|            lcnet_050            | 128 |  1.9732  |
|        convmixer_768_32         | 32  |  1.9472  |
|           selecsls42b           | 128 |  1.8057  |
|           resnest101e           | 32  |  1.7945  |
|          botnet26t_256          |  0  |   nan    |
|       eca_botnext26ts_256       |  0  |   nan    |
|        eca_halonext26ts         |  0  |   nan    |
|        sebotnet33ts_256         |  0  |   nan    |
+---------------------------------+-----+----------+

Peak Memory Compression Ratio

+---------------------------------+-----+----------+
|              name               | bs  | inductor |
+---------------------------------+-----+----------+
|        ese_vovnet19b_dw         | 128 |  0.9974  |
|           dm_nfnet_f0           | 128 |  0.9972  |
|      beit_base_patch16_224      | 64  |  0.9967  |
| deit_base_distilled_patch16_224 | 64  |  0.9965  |
|          mixer_b16_224          | 64  |  0.9963  |
|            gernet_l             | 128 |  0.9963  |
|            fbnetv3_b            | 128 |  0.9962  |
|            pit_b_224            | 64  |  0.9962  |
|            nfnet_l0             | 64  |  0.9962  |
|          cspdarknet53           | 64  |  0.9961  |
|        gluon_xception65         | 32  |  0.9955  |
|          resmlp_12_224          | 128 |  0.9951  |
|          inception_v3           | 128 |  0.9945  |
|            repvgg_a2            | 128 |  0.9943  |
|          convnext_base          | 32  |  0.9942  |
|           rexnet_100            | 128 |  0.9942  |
|         poolformer_m36          | 64  |  0.9937  |
|             dla102              | 64  |  0.9929  |
|           tf_mixnet_l           | 64  |  0.9928  |
|           mobilevit_s           | 32  |  0.9926  |
|          pnasnet5large          | 16  |  0.9926  |
|           fbnetc_100            | 128 |  0.992   |
|       tf_efficientnet_b0        | 128 |  0.9919  |
|           selecsls42b           | 128 |  0.9914  |
|           volo_d1_224           | 64  |  0.9913  |
|     swsl_resnext101_32x16d      | 32  |  0.9912  |
|            mixnet_l             | 64  |  0.9895  |
|          jx_nest_base           | 32  |  0.9894  |
|      xcit_large_24_p8_224       |  5  |  0.9892  |
|          cait_m36_384           |  2  |  0.9883  |
|        res2net101_26w_4s        | 64  |  0.9867  |
|        adv_inception_v3         | 128 |  0.9863  |
|       gluon_inception_v3        | 128 |  0.9855  |
|           mnasnet_100           | 128 |  0.9852  |
|          spnasnet_100           | 128 |  0.9852  |
|          ghostnet_100           | 128 |  0.9845  |
|            lcnet_050            | 128 |  0.9831  |
|           regnety_002           | 128 |  0.9808  |
|            tinynet_a            | 128 |  0.9741  |
|        convmixer_768_32         | 32  |  0.9715  |
|           res2next50            |  2  |  0.9548  |
|        res2net50_14w_8s         |  2  |  0.9372  |
|          gmlp_s16_224           | 64  |  0.9283  |
|          gmixer_24_224          | 64  |  0.9094  |
|        twins_pcpvt_base         | 32  |  0.9033  |
|  swin_base_patch4_window7_224   | 64  |  0.9009  |
|           resnest101e           | 32  |  0.8544  |
|      mobilenetv3_large_100      | 128 |  0.8465  |
|           convit_base           | 32  |  0.8249  |
|            hrnet_w18            |  2  |  0.8205  |
|         coat_lite_mini          | 128 |  0.8071  |
|         mobilenetv2_100         | 128 |  0.805   |
|      vit_base_patch16_224       | 64  |  0.8042  |
|         visformer_small         | 128 |  0.7898  |
|             dpn107              | 32  |  0.7697  |
|         crossvit_9_240          | 64  |  0.7352  |
|        tnt_s_patch16_224        | 64  |  0.6241  |
|          botnet26t_256          |  0  |   nan    |
|       eca_botnext26ts_256       |  0  |   nan    |
|        eca_halonext26ts         |  0  |   nan    |
|        sebotnet33ts_256         |  0  |   nan    |
+---------------------------------+-----+----------+

[yudongsi]

Performance Dashboard for float32 precision -- Single-core Single-thread

Executive Summary

see more

We evaluate different backends across three benchmark suites - torchbench, huggingface and timm. We run these experiments on Intel(R) Xeon(R) Platinum 8358 CPU @ 2.60GHz. Each experiment runs one iteration of forward pass. For accuracy, we check the numerical correctness of forward pass outputs by comparing with native pytorch. We measure speedup by normalizing against the performance of native pytorch. We report mean compilation latency numbers and peak memory footprint reduction ratio.

Caveats

1. Batch size has been reduced to workaround OOM errors. Work is in progress to reduce peak memory footprint.
2. Experiments do not cover dynamic shapes.
3. Experimental setup does not have optimizer.

To measure performance, compilation latency and memory footprint reduction, we remove the models that fail accuracy checks.

Passrate

+----------+------------+--------------+-------------+
| Compiler | torchbench | huggingface  | timm_models |
+----------+------------+--------------+-------------+
| inductor | 91%, 49/54 |  95%, 42/44  |  90%, 55/61 |  
+----------+------------+--------------+-------------+

Geometric mean speedup

+----------+------------+--------------+-------------+
| Compiler | torchbench | huggingface  | timm_models |
+----------+------------+--------------+-------------+
| inductor |   1.03x    |    1.00x     |    1.01x    |  
+----------+------------+--------------+-------------+

Mean compilation time (seconds)

+----------+------------+--------------+-------------+
| Compiler | torchbench | huggingface  | timm_models |
+----------+------------+--------------+-------------+
| inductor |   5.47     |     7.35     |    5.69     |  
+----------+------------+--------------+-------------+

Peak memory footprint compression ratio (higher is better)

+----------+------------+--------------+-------------+
| Compiler | torchbench | huggingface  | timm_models |
+----------+------------+--------------+-------------+
| inductor |   0.97x    |    0.98x     |     0.95x   |  
+----------+------------+--------------+-------------+

torchbench suite with float32 precision

see more

Performance speedup

+-----------------------------------+-----+----------+
|               name                | bs  | inductor |
+-----------------------------------+-----+----------+
|        shufflenet_v2_x1_0         |  1  |  1.3718  |
|           squeezenet1_1           |  1  |  1.3473  |
|   pytorch_CycleGAN_and_pix2pix    |  1  |  1.1701  |
|           timm_resnest            |  1  |  1.162   |
|             resnet18              |  1  |  1.121   |
| attention_is_all_you_need_pytorch |  1  |  1.0937  |
|              alexnet              |  1  |  1.0816  |
|               vgg16               |  1  |  1.0685  |
|            Super_SloMo            |  1  |  1.055   |
|                drq                |  1  |  1.0498  |
|         soft_actor_critic         | 256 |  1.0109  |
|               dcgan               |  1  |  1.007   |
|              demucs               |  1  |  1.0053  |
|               dlrm                |  1  |  1.0047  |
|          pytorch_stargan          | 16  |  0.9986  |
|            tts_angular            |  1  |  0.9985  |
|          LearningToPaint          |  1  |  0.9972  |
|      resnet50_quantized_qat       |  1  |  0.9955  |
|    mobilenet_v2_quantized_qat     |  1  |  0.9899  |
|             resnet50              |  1  |  0.9861  |
|            hf_BigBird             |  1  |  0.9834  |
|      nvidia_deeprecommender       |  1  |  0.9787  |
|            timm_nfnet             |  1  |  0.9738  |
|           pytorch_unet            |  1  |  0.9646  |
|        Background_Matting         |  1  |  0.951   |
|            timm_vovnet            |  1  |  0.9484  |
|            timm_regnet            |  1  |  0.9359  |
|        speech_transformer         |  1  |  0.9162  |
|          resnext50_32x4d          |  1  |  0.8816  |
|           hf_Longformer           |  1  |  0.8638  |
|              hf_GPT2              |  1  |  0.8593  |
|            mnasnet1_0             |  1  |  0.8526  |
|            hf_Reformer            |  1  |  0.8378  |
|          vision_maskrcnn          |  1  |  0.8344  |
|              yolov3               |  1  |  0.8325  |
|           BERT_pytorch            |  1  |  0.7956  |
|            hf_T5_large            |  1  |  0.7947  |
|   timm_vision_transformer_large   |  1  |  0.7892  |
|             hf_Albert             |  1  |  0.7764  |
|        mobilenet_v3_large         |  1  |  0.7747  |
|            densenet121            |  1  |  0.7624  |
|           lennard_jones           |  1  |  0.7559  |
|               hf_T5               |  1  |  0.7424  |
|           hf_GPT2_large           |  1  |  0.7353  |
|           hf_DistilBert           |  1  |  0.7248  |
|              hf_Bert              |  1  |  0.6891  |
|              hf_Bart              |  1  |  0.6826  |
|           fastNLP_Bert            |  1  |  0.6784  |
|            hf_T5_base             |  1  |  0.6702  |
|           mobilenet_v2            |  1  |  0.6445  |
|      timm_vision_transformer      |  1  |  0.5835  |
|       functorch_dp_cifar10        |  1  |  0.5038  |
|         timm_efficientnet         |  1  |  0.4167  |
|     detectron2_fcos_r_50_fpn      |  0  |   0.0    |
+-----------------------------------+-----+----------+

Accuracy

+-----------------------------------+-----+------------------+
|               name                | bs  |     inductor     |
+-----------------------------------+-----+------------------+
|            hf_T5_large            |  1  | pass_due_to_skip |
|   timm_vision_transformer_large   |  1  | pass_due_to_skip |
|           hf_GPT2_large           |  1  | pass_due_to_skip |
|           BERT_pytorch            |  1  |       pass       |
|        shufflenet_v2_x1_0         |  1  |       pass       |
|        mobilenet_v3_large         |  1  |       pass       |
|      nvidia_deeprecommender       |  1  |       pass       |
|   pytorch_CycleGAN_and_pix2pix    |  1  |       pass       |
|          pytorch_stargan          | 16  |       pass       |
|           pytorch_unet            |  1  |       pass       |
|             resnet18              |  1  |       pass       |
|             resnet50              |  1  |       pass       |
|      resnet50_quantized_qat       |  1  |       pass       |
|          resnext50_32x4d          |  1  |       pass       |
|         soft_actor_critic         | 256 |       pass       |
|           mobilenet_v2            |  1  |       pass       |
|        speech_transformer         |  1  |       pass       |
|           squeezenet1_1           |  1  |       pass       |
|         timm_efficientnet         |  1  |       pass       |
|            timm_nfnet             |  1  |       pass       |
|            timm_regnet            |  1  |       pass       |
|           timm_resnest            |  1  |       pass       |
|      timm_vision_transformer      |  1  |       pass       |
|            timm_vovnet            |  1  |       pass       |
|            tts_angular            |  1  |       pass       |
|               vgg16               |  1  |       pass       |
|    mobilenet_v2_quantized_qat     |  1  |       pass       |
|            mnasnet1_0             |  1  |       pass       |
|           fastNLP_Bert            |  1  |       pass       |
|       functorch_dp_cifar10        |  1  |       pass       |
|          LearningToPaint          |  1  |       pass       |
|            Super_SloMo            |  1  |       pass       |
|              alexnet              |  1  |       pass       |
| attention_is_all_you_need_pytorch |  1  |       pass       |
|               dcgan               |  1  |       pass       |
|              demucs               |  1  |       pass       |
|            densenet121            |  1  |       pass       |
|               dlrm                |  1  |       pass       |
|                drq                |  1  |       pass       |
|           lennard_jones           |  1  |       pass       |
|              yolov3               |  1  |       pass       |
|             hf_Albert             |  1  |       pass       |
|              hf_Bart              |  1  |       pass       |
|              hf_Bert              |  1  |       pass       |
|            hf_BigBird             |  1  |       pass       |
|           hf_Longformer           |  1  |       pass       |
|            hf_Reformer            |  1  |       pass       |
|               hf_T5               |  1  |       pass       |
|            hf_T5_base             |  1  |       pass       |
|        Background_Matting         |  1  |  fail_accuracy   |
|     detectron2_fcos_r_50_fpn      |  0  |      0.0000      |
|           hf_DistilBert           |  0  |      0.0000      |
|              hf_GPT2              |  0  |      0.0000      |
|          vision_maskrcnn          |  0  |      0.0000      |
+-----------------------------------+-----+------------------+

Compilation latency (sec)

+-----------------------------------+-----+----------+
|               name                | bs  | inductor |
+-----------------------------------+-----+----------+
|            hf_T5_base             |  1  | 50.6758  |
|           hf_GPT2_large           |  1  |  34.583  |
|            hf_T5_large            |  1  | 28.4916  |
|          vision_maskrcnn          |  1  | 18.0325  |
|   timm_vision_transformer_large   |  1  | 13.3261  |
|           hf_Longformer           |  1  | 12.2457  |
|            hf_BigBird             |  1  | 10.1571  |
|              yolov3               |  1  |  9.6826  |
|            densenet121            |  1  |  7.9872  |
|              hf_Bart              |  1  |  6.8012  |
|            timm_nfnet             |  1  |  6.7634  |
|           fastNLP_Bert            |  1  |  6.5982  |
|              hf_Bert              |  1  |  5.2685  |
|               hf_T5               |  1  |  5.0497  |
|        speech_transformer         |  1  |  4.7929  |
|        Background_Matting         |  1  |  4.684   |
|            timm_regnet            |  1  |  4.2927  |
|            Super_SloMo            |  1  |  4.2702  |
|             hf_Albert             |  1  |  3.9343  |
|            hf_Reformer            |  1  |  3.7558  |
|           BERT_pytorch            |  1  |  3.7028  |
|        shufflenet_v2_x1_0         |  1  |  3.6925  |
|              hf_GPT2              |  1  |  3.5584  |
|         timm_efficientnet         |  1  |  3.4965  |
| attention_is_all_you_need_pytorch |  1  |  3.3817  |
|             resnet50              |  1  |  2.8523  |
|      timm_vision_transformer      |  1  |  2.8507  |
|          resnext50_32x4d          |  1  |  2.8399  |
|            mnasnet1_0             |  1  |  2.7662  |
|        mobilenet_v3_large         |  1  |  2.7274  |
|           pytorch_unet            |  1  |  2.7155  |
|           mobilenet_v2            |  1  |  2.6687  |
|            timm_vovnet            |  1  |  2.546   |
|           hf_DistilBert           |  1  |  2.4877  |
|           timm_resnest            |  1  |  2.2898  |
|       functorch_dp_cifar10        |  1  |  1.917   |
|          pytorch_stargan          | 16  |  1.3333  |
|          LearningToPaint          |  1  |  1.0428  |
|             resnet18              |  1  |  0.9963  |
|   pytorch_CycleGAN_and_pix2pix    |  1  |  0.8937  |
|               dlrm                |  1  |  0.8525  |
|              demucs               |  1  |  0.816   |
|           squeezenet1_1           |  1  |  0.6268  |
|            tts_angular            |  1  |  0.5151  |
|               vgg16               |  1  |  0.3717  |
|              alexnet              |  1  |  0.3331  |
|                drq                |  1  |  0.3052  |
|               dcgan               |  1  |  0.298   |
|      nvidia_deeprecommender       |  1  |  0.2661  |
|         soft_actor_critic         | 256 |  0.2076  |
|           lennard_jones           |  1  |  0.1296  |
|    mobilenet_v2_quantized_qat     |  1  |  0.0764  |
|      resnet50_quantized_qat       |  1  |  0.0556  |
|     detectron2_fcos_r_50_fpn      |  0  |   nan    |
+-----------------------------------+-----+----------+

Peak Memory Compression Ratio

+-----------------------------------+-----+----------+
|               name                | bs  | inductor |
+-----------------------------------+-----+----------+
|              demucs               |  1  |  0.999   |
|      nvidia_deeprecommender       |  1  |  0.9966  |
|      resnet50_quantized_qat       |  1  |  0.9953  |
|               vgg16               |  1  |  0.9952  |
|            hf_T5_base             |  1  |  0.9948  |
|        Background_Matting         |  1  |  0.9946  |
|    mobilenet_v2_quantized_qat     |  1  |  0.993   |
|           hf_GPT2_large           |  1  |  0.9927  |
|              alexnet              |  1  |  0.9917  |
|            hf_BigBird             |  1  |  0.9915  |
|           pytorch_unet            |  1  |  0.9913  |
|         soft_actor_critic         | 256 |  0.9909  |
|          pytorch_stargan          | 16  |  0.9906  |
|          LearningToPaint          |  1  |  0.9905  |
|   timm_vision_transformer_large   |  1  |  0.9901  |
|            tts_angular            |  1  |  0.9876  |
|           lennard_jones           |  1  |  0.9871  |
|                drq                |  1  |  0.9868  |
|               dcgan               |  1  |  0.9846  |
| attention_is_all_you_need_pytorch |  1  |  0.9835  |
|              hf_GPT2              |  1  |  0.9834  |
|           fastNLP_Bert            |  1  |  0.9824  |
|            hf_T5_large            |  1  |  0.9816  |
|           hf_DistilBert           |  1  |  0.9815  |
|              hf_Bert              |  1  |  0.9815  |
|           BERT_pytorch            |  1  |  0.9808  |
|        speech_transformer         |  1  |  0.9796  |
|   pytorch_CycleGAN_and_pix2pix    |  1  |  0.9788  |
|             hf_Albert             |  1  |  0.9783  |
|               hf_T5               |  1  |  0.9752  |
|             resnet18              |  1  |  0.9752  |
|          vision_maskrcnn          |  1  |  0.975   |
|              hf_Bart              |  1  |  0.973   |
|            Super_SloMo            |  1  |  0.973   |
|           hf_Longformer           |  1  |  0.9685  |
|           timm_resnest            |  1  |  0.968   |
|            timm_vovnet            |  1  |  0.9678  |
|            timm_regnet            |  1  |  0.9662  |
|          resnext50_32x4d          |  1  |  0.9637  |
|             resnet50              |  1  |  0.9636  |
|           squeezenet1_1           |  1  |  0.9626  |
|            timm_nfnet             |  1  |  0.9595  |
|      timm_vision_transformer      |  1  |  0.9577  |
|              yolov3               |  1  |  0.9573  |
|               dlrm                |  1  |  0.9548  |
|            mnasnet1_0             |  1  |  0.9511  |
|        mobilenet_v3_large         |  1  |  0.9489  |
|           mobilenet_v2            |  1  |  0.9488  |
|       functorch_dp_cifar10        |  1  |  0.9474  |
|            hf_Reformer            |  1  |  0.9454  |
|        shufflenet_v2_x1_0         |  1  |  0.9432  |
|         timm_efficientnet         |  1  |  0.943   |
|            densenet121            |  1  |  0.8713  |
|     detectron2_fcos_r_50_fpn      |  0  |   nan    |
+-----------------------------------+-----+----------+

huggingface suite with float32 precision

see more

Performance speedup

+-----------------------------------------+----+----------+
|                  name                   | bs | inductor |
+-----------------------------------------+----+----------+
|     MobileBertForQuestionAnswering      | 1  |  1.0345  |
|            XLNetLMHeadModel             | 1  |  0.9686  |
|            AlbertForMaskedLM            | 1  |  0.9365  |
|       AlbertForQuestionAnswering        | 1  |  0.932   |
|          MobileBertForMaskedLM          | 1  |  0.8867  |
|                 BigBird                 | 1  |  0.8861  |
|     M2M100ForConditionalGeneration      | 1  |  0.8828  |
|             OPTForCausalLM              | 1  |  0.8575  |
|               GoogleFnet                | 1  |  0.8302  |
|       DebertaForQuestionAnswering       | 1  |  0.7953  |
|          AllenaiLongformerBase          | 1  |  0.792   |
|         MegatronBertForCausalLM         | 1  |  0.7858  |
|            YituTechConvBert             | 1  |  0.7855  |
|    MegatronBertForQuestionAnswering     | 1  |  0.7855  |
|     PegasusForConditionalGeneration     | 1  |  0.7852  |
|         Speech2Text2ForCausalLM         | 1  |  0.7839  |
|      MBartForConditionalGeneration      | 1  |  0.7785  |
|            TrOCRForCausalLM             | 1  |  0.7713  |
|           PegasusForCausalLM            | 1  |  0.7688  |
|            MBartForCausalLM             | 1  |  0.7678  |
|             XGLMForCausalLM             | 1  |  0.7664  |
|       RobertaForQuestionAnswering       | 1  |  0.7628  |
|           DebertaForMaskedLM            | 1  |  0.7606  |
|     DistilBertForQuestionAnswering      | 1  |  0.7468  |
|           RobertaForCausalLM            | 1  |  0.7448  |
|     PLBartForConditionalGeneration      | 1  |  0.7417  |
|          DistilBertForMaskedLM          | 1  |  0.7257  |
|            PLBartForCausalLM            | 1  |  0.7234  |
|               DistillGPT2               | 1  |  0.7063  |
|           LayoutLMForMaskedLM           | 1  |  0.6533  |
|    LayoutLMForSequenceClassification    | 1  |  0.6507  |
|             BartForCausalLM             | 1  |  0.6452  |
|      GPT2ForSequenceClassification      | 1  |  0.645   |
|                CamemBert                | 1  |  0.6426  |
| BlenderbotSmallForConditionalGeneration | 1  |  0.6426  |
|       BlenderbotSmallForCausalLM        | 1  |  0.6418  |
|       T5ForConditionalGeneration        | 1  |  0.6012  |
|                 T5Small                 | 1  |  0.6006  |
|      BartForConditionalGeneration       | 1  |  0.592   |
|       ElectraForQuestionAnswering       | 1  |  0.4796  |
|           ElectraForCausalLM            | 1  |  0.4554  |
|       MT5ForConditionalGeneration       | 1  |  0.4265  |
|             BertForMaskedLM             | 0  |   0.0    |
|        BertForQuestionAnswering         | 0  |   0.0    |
+-----------------------------------------+----+----------+

Accuracy

+-----------------------------------------+----+----------+
|                  name                   | bs | inductor |
+-----------------------------------------+----+----------+
|            AlbertForMaskedLM            | 1  |   pass   |
|       AlbertForQuestionAnswering        | 1  |   pass   |
|      MBartForConditionalGeneration      | 1  |   pass   |
|       MT5ForConditionalGeneration       | 1  |   pass   |
|         MegatronBertForCausalLM         | 1  |   pass   |
|    MegatronBertForQuestionAnswering     | 1  |   pass   |
|          MobileBertForMaskedLM          | 1  |   pass   |
|     MobileBertForQuestionAnswering      | 1  |   pass   |
|             OPTForCausalLM              | 1  |   pass   |
|            PLBartForCausalLM            | 1  |   pass   |
|     PLBartForConditionalGeneration      | 1  |   pass   |
|           PegasusForCausalLM            | 1  |   pass   |
|     PegasusForConditionalGeneration     | 1  |   pass   |
|           RobertaForCausalLM            | 1  |   pass   |
|       RobertaForQuestionAnswering       | 1  |   pass   |
|         Speech2Text2ForCausalLM         | 1  |   pass   |
|       T5ForConditionalGeneration        | 1  |   pass   |
|                 T5Small                 | 1  |   pass   |
|            TrOCRForCausalLM             | 1  |   pass   |
|             XGLMForCausalLM             | 1  |   pass   |
|            XLNetLMHeadModel             | 1  |   pass   |
|            MBartForCausalLM             | 1  |   pass   |
|     M2M100ForConditionalGeneration      | 1  |   pass   |
|    LayoutLMForSequenceClassification    | 1  |   pass   |
|                CamemBert                | 1  |   pass   |
|          AllenaiLongformerBase          | 1  |   pass   |
|             BartForCausalLM             | 1  |   pass   |
|      BartForConditionalGeneration       | 1  |   pass   |
|             BertForMaskedLM             | 1  |   pass   |
|        BertForQuestionAnswering         | 1  |   pass   |
|                 BigBird                 | 1  |   pass   |
|       BlenderbotSmallForCausalLM        | 1  |   pass   |
| BlenderbotSmallForConditionalGeneration | 1  |   pass   |
|           DebertaForMaskedLM            | 1  |   pass   |
|           LayoutLMForMaskedLM           | 1  |   pass   |
|       DebertaForQuestionAnswering       | 1  |   pass   |
|          DistilBertForMaskedLM          | 1  |   pass   |
|     DistilBertForQuestionAnswering      | 1  |   pass   |
|               DistillGPT2               | 1  |   pass   |
|           ElectraForCausalLM            | 1  |   pass   |
|       ElectraForQuestionAnswering       | 1  |   pass   |
|      GPT2ForSequenceClassification      | 1  |   pass   |
|               GoogleFnet                | 1  |   pass   |
|            YituTechConvBert             | 1  |   pass   |
+-----------------------------------------+----+----------+

Compilation latency (sec)

+-----------------------------------------+----+----------+
|                  name                   | bs | inductor |
+-----------------------------------------+----+----------+
|      BartForConditionalGeneration       | 1  | 31.7606  |
|          AllenaiLongformerBase          | 1  | 15.2198  |
|             BartForCausalLM             | 1  | 12.4113  |
|     MobileBertForQuestionAnswering      | 1  | 11.9979  |
|          MobileBertForMaskedLM          | 1  | 11.9653  |
|                 BigBird                 | 1  | 11.1688  |
|            XLNetLMHeadModel             | 1  | 10.6341  |
|           DebertaForMaskedLM            | 1  | 10.3189  |
|      MBartForConditionalGeneration      | 1  |  9.6144  |
|       DebertaForQuestionAnswering       | 1  |  9.5465  |
|     PegasusForConditionalGeneration     | 1  |  9.3128  |
|                 T5Small                 | 1  |  9.1569  |
|       T5ForConditionalGeneration        | 1  |  9.1361  |
|             XGLMForCausalLM             | 1  |  8.3606  |
|     M2M100ForConditionalGeneration      | 1  |  8.2767  |
|            AlbertForMaskedLM            | 1  |  8.2582  |
|      GPT2ForSequenceClassification      | 1  |  8.1982  |
|       AlbertForQuestionAnswering        | 1  |  8.1664  |
|         MegatronBertForCausalLM         | 1  |  7.8404  |
|    MegatronBertForQuestionAnswering     | 1  |  7.6892  |
|       MT5ForConditionalGeneration       | 1  |  7.571   |
|            YituTechConvBert             | 1  |   6.62   |
|                CamemBert                | 1  |  6.0292  |
| BlenderbotSmallForConditionalGeneration | 1  |  5.8922  |
|           LayoutLMForMaskedLM           | 1  |  5.7682  |
|    LayoutLMForSequenceClassification    | 1  |  5.3124  |
|     PLBartForConditionalGeneration      | 1  |  4.6582  |
|           ElectraForCausalLM            | 1  |  4.6399  |
|       ElectraForQuestionAnswering       | 1  |  4.2141  |
|           PegasusForCausalLM            | 1  |  4.0619  |
|            MBartForCausalLM             | 1  |  4.0019  |
|            TrOCRForCausalLM             | 1  |  3.8745  |
|           RobertaForCausalLM            | 1  |  3.7961  |
|       RobertaForQuestionAnswering       | 1  |  3.6619  |
|             OPTForCausalLM              | 1  |  3.2093  |
|               DistillGPT2               | 1  |  3.0738  |
|               GoogleFnet                | 1  |  2.7978  |
|       BlenderbotSmallForCausalLM        | 1  |  2.5215  |
|         Speech2Text2ForCausalLM         | 1  |   2.15   |
|            PLBartForCausalLM            | 1  |  1.9917  |
|          DistilBertForMaskedLM          | 1  |  1.9543  |
|     DistilBertForQuestionAnswering      | 1  |  1.9133  |
|             BertForMaskedLM             | 0  |   nan    |
|        BertForQuestionAnswering         | 0  |   nan    |
+-----------------------------------------+----+----------+

Peak Memory Compression Ratio

+-----------------------------------------+----+----------+
|                  name                   | bs | inductor |
+-----------------------------------------+----+----------+
|            AlbertForMaskedLM            | 1  |  0.9949  |
|       AlbertForQuestionAnswering        | 1  |  0.9948  |
|                 BigBird                 | 1  |  0.9932  |
|               GoogleFnet                | 1  |  0.9915  |
|      BartForConditionalGeneration       | 1  |  0.9905  |
|      GPT2ForSequenceClassification      | 1  |  0.9904  |
|           DebertaForMaskedLM            | 1  |  0.9886  |
|     PegasusForConditionalGeneration     | 1  |  0.9885  |
|       DebertaForQuestionAnswering       | 1  |  0.9879  |
|            TrOCRForCausalLM             | 1  |  0.9876  |
|     M2M100ForConditionalGeneration      | 1  |  0.9874  |
|            MBartForCausalLM             | 1  |  0.9858  |
|       MT5ForConditionalGeneration       | 1  |  0.985   |
|               DistillGPT2               | 1  |  0.9846  |
|                 T5Small                 | 1  |  0.9841  |
|       T5ForConditionalGeneration        | 1  |  0.9841  |
|             XGLMForCausalLM             | 1  |  0.9831  |
|           PegasusForCausalLM            | 1  |  0.9827  |
|            XLNetLMHeadModel             | 1  |  0.9824  |
|    LayoutLMForSequenceClassification    | 1  |  0.9821  |
|           LayoutLMForMaskedLM           | 1  |  0.9818  |
|     DistilBertForQuestionAnswering      | 1  |  0.9817  |
|          DistilBertForMaskedLM          | 1  |  0.9817  |
|         MegatronBertForCausalLM         | 1  |  0.9815  |
|                CamemBert                | 1  |  0.9815  |
|    MegatronBertForQuestionAnswering     | 1  |  0.9811  |
|           RobertaForCausalLM            | 1  |  0.9794  |
|       RobertaForQuestionAnswering       | 1  |  0.9792  |
|          AllenaiLongformerBase          | 1  |  0.979   |
|      MBartForConditionalGeneration      | 1  |  0.9778  |
|             OPTForCausalLM              | 1  |  0.9769  |
|            YituTechConvBert             | 1  |  0.976   |
|     PLBartForConditionalGeneration      | 1  |  0.974   |
|       BlenderbotSmallForCausalLM        | 1  |  0.9736  |
|         Speech2Text2ForCausalLM         | 1  |  0.9733  |
|            PLBartForCausalLM            | 1  |  0.9681  |
|           ElectraForCausalLM            | 1  |  0.9627  |
|             BartForCausalLM             | 1  |  0.9607  |
|       ElectraForQuestionAnswering       | 1  |  0.958   |
| BlenderbotSmallForConditionalGeneration | 1  |  0.9569  |
|          MobileBertForMaskedLM          | 1  |  0.9308  |
|     MobileBertForQuestionAnswering      | 1  |  0.9129  |
|             BertForMaskedLM             | 0  |   nan    |
|        BertForQuestionAnswering         | 0  |   nan    |
+-----------------------------------------+----+----------+

timm_models suite with float32 precision

see more

Performance speedup

+---------------------------------+----+----------+
|              name               | bs | inductor |
+---------------------------------+----+----------+
|          pnasnet5large          | 1  |  1.1504  |
|          inception_v3           | 1  |  1.1191  |
|       gluon_inception_v3        | 1  |  1.0991  |
|        adv_inception_v3         | 1  |  1.098   |
|        res2net101_26w_4s        | 1  |  1.0485  |
|          ghostnet_100           | 1  |  1.039   |
|        res2net50_14w_8s         | 1  |  1.0334  |
|           resnest101e           | 1  |  1.0113  |
|            nfnet_l0             | 1  |  1.0066  |
|            hrnet_w18            | 1  |  1.0041  |
|           regnety_002           | 1  |  0.9546  |
|            repvgg_a2            | 1  |  0.9453  |
|           selecsls42b           | 1  |  0.9205  |
|           dm_nfnet_f0           | 1  |  0.9179  |
|             dpn107              | 1  |  0.9178  |
|          spnasnet_100           | 1  |  0.9163  |
|           mnasnet_100           | 1  |  0.9118  |
|           res2next50            | 1  |  0.8947  |
|            gernet_l             | 1  |  0.8872  |
|     swsl_resnext101_32x16d      | 1  |  0.8771  |
|          gmixer_24_224          | 1  |  0.8391  |
|      xcit_large_24_p8_224       | 1  |  0.8366  |
|        convmixer_768_32         | 1  |  0.8288  |
|      beit_base_patch16_224      | 1  |  0.8284  |
|             dla102              | 1  |  0.8054  |
|      mobilenetv3_large_100      | 1  |  0.8004  |
|        gluon_xception65         | 1  |  0.7888  |
|           fbnetc_100            | 1  |  0.7678  |
| deit_base_distilled_patch16_224 | 1  |  0.7665  |
|      vit_base_patch16_224       | 1  |  0.7505  |
|            fbnetv3_b            | 1  |  0.7415  |
|           volo_d1_224           | 1  |  0.7371  |
|           convit_base           | 1  |  0.7311  |
|  swin_base_patch4_window7_224   | 1  |  0.7262  |
|          cait_m36_384           | 1  |  0.7251  |
|          mixer_b16_224          | 1  |  0.7156  |
|            lcnet_050            | 1  |  0.7139  |
|         poolformer_m36          | 1  |  0.6751  |
|         mobilenetv2_100         | 1  |  0.6626  |
|          cspdarknet53           | 1  |  0.6616  |
|          resmlp_12_224          | 1  |   0.66   |
|            pit_b_224            | 1  |  0.6482  |
|          convnext_base          | 1  |  0.6364  |
|        tnt_s_patch16_224        | 1  |  0.6352  |
|        twins_pcpvt_base         | 1  |  0.6059  |
|          jx_nest_base           | 1  |  0.6047  |
|         visformer_small         | 1  |  0.6011  |
|         crossvit_9_240          | 1  |  0.5865  |
|           tf_mixnet_l           | 1  |  0.5648  |
|        ese_vovnet19b_dw         | 1  |  0.546   |
|            mixnet_l             | 1  |  0.5282  |
|         coat_lite_mini          | 1  |  0.5237  |
|           rexnet_100            | 1  |  0.4773  |
|            tinynet_a            | 1  |  0.4549  |
|       tf_efficientnet_b0        | 1  |  0.4062  |
|           mobilevit_s           | 1  |  0.3664  |
|          gmlp_s16_224           | 1  |  0.364   |
|        sebotnet33ts_256         | 0  |   0.0    |
|          botnet26t_256          | 0  |   0.0    |
|       eca_botnext26ts_256       | 0  |   0.0    |
|        eca_halonext26ts         | 0  |   0.0    |
+---------------------------------+----+----------+

Accuracy

+---------------------------------+----+---------------+
|              name               | bs |   inductor    |
+---------------------------------+----+---------------+
|        adv_inception_v3         | 1  |     pass      |
|      beit_base_patch16_224      | 1  |     pass      |
|         mobilenetv2_100         | 1  |     pass      |
|      mobilenetv3_large_100      | 1  |     pass      |
|           mobilevit_s           | 1  |     pass      |
|            nfnet_l0             | 1  |     pass      |
|            pit_b_224            | 1  |     pass      |
|          pnasnet5large          | 1  |     pass      |
|         poolformer_m36          | 1  |     pass      |
|           regnety_002           | 1  |     pass      |
|            repvgg_a2            | 1  |     pass      |
|        res2net101_26w_4s        | 1  |     pass      |
|        res2net50_14w_8s         | 1  |     pass      |
|           res2next50            | 1  |     pass      |
|          resmlp_12_224          | 1  |     pass      |
|           resnest101e           | 1  |     pass      |
|           rexnet_100            | 1  |     pass      |
|           selecsls42b           | 1  |     pass      |
|          spnasnet_100           | 1  |     pass      |
|  swin_base_patch4_window7_224   | 1  |     pass      |
|     swsl_resnext101_32x16d      | 1  |     pass      |
|       tf_efficientnet_b0        | 1  |     pass      |
|           tf_mixnet_l           | 1  |     pass      |
|            tinynet_a            | 1  |     pass      |
|        tnt_s_patch16_224        | 1  |     pass      |
|        twins_pcpvt_base         | 1  |     pass      |
|         visformer_small         | 1  |     pass      |
|      vit_base_patch16_224       | 1  |     pass      |
|           volo_d1_224           | 1  |     pass      |
|           mnasnet_100           | 1  |     pass      |
|            mixnet_l             | 1  |     pass      |
|          mixer_b16_224          | 1  |     pass      |
|        ese_vovnet19b_dw         | 1  |     pass      |
|         coat_lite_mini          | 1  |     pass      |
|           convit_base           | 1  |     pass      |
|        convmixer_768_32         | 1  |     pass      |
|          convnext_base          | 1  |     pass      |
|         crossvit_9_240          | 1  |     pass      |
|          cspdarknet53           | 1  |     pass      |
| deit_base_distilled_patch16_224 | 1  |     pass      |
|             dla102              | 1  |     pass      |
|           dm_nfnet_f0           | 1  |     pass      |
|             dpn107              | 1  |     pass      |
|            lcnet_050            | 1  |     pass      |
|      xcit_large_24_p8_224       | 1  |     pass      |
|           fbnetc_100            | 1  |     pass      |
|            fbnetv3_b            | 1  |     pass      |
|            gernet_l             | 1  |     pass      |
|       gluon_inception_v3        | 1  |     pass      |
|        gluon_xception65         | 1  |     pass      |
|          gmixer_24_224          | 1  |     pass      |
|          gmlp_s16_224           | 1  |     pass      |
|            hrnet_w18            | 1  |     pass      |
|          inception_v3           | 1  |     pass      |
|          jx_nest_base           | 1  |     pass      |
|        eca_halonext26ts         | 1  |  fail_to_run  |
|        sebotnet33ts_256         | 1  |  fail_to_run  |
|          botnet26t_256          | 1  |  fail_to_run  |
|       eca_botnext26ts_256       | 1  |  fail_to_run  |
|          ghostnet_100           | 1  | fail_accuracy |
|          cait_m36_384           | 1  | fail_accuracy |
+---------------------------------+----+---------------+

Compilation latency (sec)

+---------------------------------+----+----------+
|              name               | bs | inductor |
+---------------------------------+----+----------+
|            hrnet_w18            | 1  |   18.0   |
|          cait_m36_384           | 1  | 17.8966  |
|          pnasnet5large          | 1  | 15.4624  |
|      xcit_large_24_p8_224       | 1  | 12.9297  |
|  swin_base_patch4_window7_224   | 1  | 11.0748  |
|         poolformer_m36          | 1  |  9.9394  |
|        res2net101_26w_4s        | 1  |  9.2803  |
|        twins_pcpvt_base         | 1  |  9.1837  |
|           mobilevit_s           | 1  |  8.716   |
|          jx_nest_base           | 1  |  8.5922  |
|        res2net50_14w_8s         | 1  |  8.3796  |
|            mixnet_l             | 1  |  8.0371  |
|           tf_mixnet_l           | 1  |  7.9076  |
|             dpn107              | 1  |  7.8182  |
|        tnt_s_patch16_224        | 1  |  7.6116  |
|          gmlp_s16_224           | 1  |  7.3036  |
|           dm_nfnet_f0           | 1  |  6.8258  |
|            fbnetv3_b            | 1  |  6.7063  |
|        gluon_xception65         | 1  |  6.5942  |
|           volo_d1_224           | 1  |  6.0258  |
|             dla102              | 1  |  5.8349  |
|          convnext_base          | 1  |  5.6419  |
|         crossvit_9_240          | 1  |  5.3923  |
|           resnest101e           | 1  |  5.2057  |
|        adv_inception_v3         | 1  |  5.1079  |
|       gluon_inception_v3        | 1  |  5.0266  |
|          inception_v3           | 1  |  4.9859  |
|           res2next50            | 1  |  4.9741  |
|          cspdarknet53           | 1  |  4.8769  |
|            nfnet_l0             | 1  |  4.5565  |
|          ghostnet_100           | 1  |  4.4579  |
|            tinynet_a            | 1  |  4.2661  |
|           rexnet_100            | 1  |  4.2428  |
|     swsl_resnext101_32x16d      | 1  |  3.9205  |
|          gmixer_24_224          | 1  |  3.8914  |
|         coat_lite_mini          | 1  |  3.8132  |
|       tf_efficientnet_b0        | 1  |  3.7351  |
|           convit_base           | 1  |  3.7254  |
|        convmixer_768_32         | 1  |  3.6167  |
|         visformer_small         | 1  |  3.6126  |
|           fbnetc_100            | 1  |  3.601   |
|          spnasnet_100           | 1  |  3.5018  |
| deit_base_distilled_patch16_224 | 1  |  3.4593  |
|            pit_b_224            | 1  |  3.4397  |
|      beit_base_patch16_224      | 1  |  3.3555  |
|            gernet_l             | 1  |  3.2903  |
|            repvgg_a2            | 1  |  3.1485  |
|      mobilenetv3_large_100      | 1  |  3.0992  |
|         mobilenetv2_100         | 1  |  3.067   |
|      vit_base_patch16_224       | 1  |  2.9249  |
|           regnety_002           | 1  |  2.8927  |
|          mixer_b16_224          | 1  |  2.8533  |
|           mnasnet_100           | 1  |  2.8398  |
|        ese_vovnet19b_dw         | 1  |  2.4235  |
|            lcnet_050            | 1  |  2.4123  |
|           selecsls42b           | 1  |  2.2144  |
|          resmlp_12_224          | 1  |  1.7754  |
|          botnet26t_256          | 0  |   nan    |
|       eca_botnext26ts_256       | 0  |   nan    |
|        eca_halonext26ts         | 0  |   nan    |
|        sebotnet33ts_256         | 0  |   nan    |
+---------------------------------+----+----------+

Peak Memory Compression Ratio

+---------------------------------+----+----------+
|              name               | bs | inductor |
+---------------------------------+----+----------+
|          cait_m36_384           | 1  |  0.983   |
|      vit_base_patch16_224       | 1  |  0.9803  |
|     swsl_resnext101_32x16d      | 1  |  0.9802  |
| deit_base_distilled_patch16_224 | 1  |  0.9785  |
|      xcit_large_24_p8_224       | 1  |  0.9776  |
|            pit_b_224            | 1  |  0.9772  |
|           selecsls42b           | 1  |  0.9742  |
|           convit_base           | 1  |  0.9734  |
|          mixer_b16_224          | 1  |  0.9731  |
|      beit_base_patch16_224      | 1  |  0.9731  |
|        convmixer_768_32         | 1  |  0.9718  |
|        ese_vovnet19b_dw         | 1  |  0.9685  |
|          convnext_base          | 1  |  0.9653  |
|            repvgg_a2            | 1  |  0.965   |
|          resmlp_12_224          | 1  |  0.9649  |
|            gernet_l             | 1  |  0.9642  |
|           dm_nfnet_f0           | 1  |  0.9628  |
|            nfnet_l0             | 1  |  0.9625  |
|            lcnet_050            | 1  |  0.9612  |
|             dpn107              | 1  |  0.9606  |
|         visformer_small         | 1  |  0.9583  |
|          cspdarknet53           | 1  |  0.9557  |
|        gluon_xception65         | 1  |  0.9537  |
|         mobilenetv2_100         | 1  |  0.9518  |
|          gmixer_24_224          | 1  |  0.9517  |
|      mobilenetv3_large_100      | 1  |  0.9515  |
|           regnety_002           | 1  |  0.9499  |
|          jx_nest_base           | 1  |  0.9497  |
|  swin_base_patch4_window7_224   | 1  |  0.9491  |
|           mnasnet_100           | 1  |  0.949   |
|         coat_lite_mini          | 1  |  0.949   |
|           res2next50            | 1  |  0.9488  |
|       tf_efficientnet_b0        | 1  |  0.9463  |
|       gluon_inception_v3        | 1  |  0.9458  |
|            tinynet_a            | 1  |  0.9453  |
|             dla102              | 1  |  0.9445  |
|        adv_inception_v3         | 1  |  0.9433  |
|          inception_v3           | 1  |  0.943   |
|          pnasnet5large          | 1  |  0.9425  |
|           fbnetc_100            | 1  |  0.9409  |
|          spnasnet_100           | 1  |  0.9397  |
|           rexnet_100            | 1  |  0.9397  |
|          ghostnet_100           | 1  |  0.9363  |
|           mobilevit_s           | 1  |  0.9355  |
|           volo_d1_224           | 1  |  0.9336  |
|           resnest101e           | 1  |  0.9329  |
|        res2net101_26w_4s        | 1  |  0.9305  |
|        res2net50_14w_8s         | 1  |  0.9252  |
|           tf_mixnet_l           | 1  |  0.9241  |
|            mixnet_l             | 1  |  0.924   |
|        tnt_s_patch16_224        | 1  |  0.9211  |
|        twins_pcpvt_base         | 1  |  0.9207  |
|         crossvit_9_240          | 1  |  0.9185  |
|            fbnetv3_b            | 1  |  0.9182  |
|         poolformer_m36          | 1  |  0.9156  |
|          gmlp_s16_224           | 1  |  0.9146  |
|            hrnet_w18            | 1  |  0.8641  |
|          botnet26t_256          | 0  |   nan    |
|       eca_botnext26ts_256       | 0  |   nan    |
|        eca_halonext26ts         | 0  |   nan    |
|        sebotnet33ts_256         | 0  |   nan    |
+---------------------------------+----+----------+

[blzheng]

Performance Dashboard for float32 precision -- Single-Socket Multi-threads

Executive Summary

see more

We evaluate different backends across three benchmark suites - torchbench, huggingface and timm. We run these experiments on Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz. Each experiment runs one iteration of forward pass. For accuracy, we check the numerical correctness of forward pass outputs by comparing with native pytorch. We measure speedup by normalizing against the performance of native pytorch. We report mean compilation latency numbers and peak memory footprint reduction ratio.

Caveats

1. Batch size has been reduced to workaround OOM errors. Work is in progress to reduce peak memory footprint.
2. Experiments do not cover dynamic shapes.
3. Experimental setup does not have optimizer.

To measure performance, compilation latency and memory footprint reduction, we remove the models that fail accuracy checks.

Passrate

+----------+------------+-------------+-------------+
| Compiler | torchbench | huggingface | timm_models |
+----------+------------+-------------+-------------+
| inductor | 94%, 51/54 | 100%, 44/44 | 90%, 55/61  |
+----------+------------+-------------+-------------+

Geometric mean speedup

+----------+------------+-------------+-------------+
| Compiler | torchbench | huggingface | timm_models |
+----------+------------+-------------+-------------+
| inductor |   1.05x    |    1.16x    |    1.05x    |
+----------+------------+-------------+-------------+

Mean compilation time (seconds)

+----------+------------+-------------+-------------+
| Compiler | torchbench | huggingface | timm_models |
+----------+------------+-------------+-------------+
| inductor |    4.21    |    6.31     |    7.17     |
+----------+------------+-------------+-------------+

Peak memory footprint compression ratio (higher is better)

+----------+------------+-------------+-------------+
| Compiler | torchbench | huggingface | timm_models |
+----------+------------+-------------+-------------+
| inductor |   0.94x    |    0.94x    |    0.98x    |
+----------+------------+-------------+-------------+

torchbench suite with float32 precision

see more

Performance speedup

+-----------------------------------+------+----------+
|               name                |  bs  | inductor |
+-----------------------------------+------+----------+
|         soft_actor_critic         | 256  |  1.6285  |
|           timm_resnest            |  32  |  1.3656  |
|             resnet50              |  32  |  1.301   |
|        shufflenet_v2_x1_0         |  64  |  1.2455  |
|            mnasnet1_0             |  32  |  1.1844  |
|                drq                |  1   |  1.146   |
|            densenet121            |  64  |  1.1441  |
|              alexnet              | 128  |  1.1201  |
|   pytorch_CycleGAN_and_pix2pix    |  1   |  1.1175  |
|           squeezenet1_1           |  16  |  1.0813  |
|               vgg16               |  4   |  1.0698  |
|               dlrm                | 2048 |  1.0623  |
|             resnet18              |  8   |  1.0495  |
|            hf_T5_large            |  1   |  1.045   |
|            Super_SloMo            |  6   |  1.0397  |
|       functorch_dp_cifar10        |  64  |  1.0378  |
|            timm_regnet            |  32  |  1.0326  |
|          pytorch_stargan          |  16  |  1.0309  |
|        mobilenet_v3_large         |  32  |  1.0179  |
|               dcgan               | 256  |  1.0161  |
|               hf_T5               |  1   |  1.0094  |
|          LearningToPaint          |  96  |  1.0057  |
|            hf_Reformer            |  1   |  1.0056  |
|      resnet50_quantized_qat       |  32  |  1.0033  |
|    mobilenet_v2_quantized_qat     |  96  |  1.001   |
|            hf_BigBird             |  1   |  1.0007  |
|              demucs               |  1   |  0.9973  |
|            tts_angular            |  64  |  0.996   |
|            timm_vovnet            |  32  |  0.9907  |
|        Background_Matting         |  1   |  0.9842  |
|           BERT_pytorch            |  2   |  0.9805  |
|           hf_GPT2_large           |  1   |  0.9283  |
|        speech_transformer         |  1   |  0.9266  |
|           mobilenet_v2            |  16  |  0.9249  |
|              hf_GPT2              |  1   |  0.9178  |
|           pytorch_unet            |  1   |  0.9163  |
|           hf_Longformer           |  1   |  0.9125  |
|             hf_Albert             |  1   |  0.8922  |
|   timm_vision_transformer_large   |  8   |  0.8638  |
|            hf_T5_base             |  1   |  0.8383  |
|      nvidia_deeprecommender       | 256  |  0.832   |
|           hf_DistilBert           |  1   |  0.8276  |
|              yolov3               |  8   |  0.8185  |
|          resnext50_32x4d          |  8   |  0.8175  |
|              hf_Bert              |  1   |  0.7978  |
|            timm_nfnet             | 128  |  0.7914  |
|              hf_Bart              |  1   |  0.7894  |
|          vision_maskrcnn          |  1   |  0.7852  |
|           fastNLP_Bert            |  1   |  0.7733  |
| attention_is_all_you_need_pytorch |  32  |  0.7196  |
|         timm_efficientnet         |  64  |  0.6824  |
|      timm_vision_transformer      |  8   |  0.681   |
|           lennard_jones           | 1000 |  0.3501  |
|     detectron2_fcos_r_50_fpn      |  0   |   0.0    |
+-----------------------------------+------+----------+

Accuracy

+-----------------------------------+-----+------------------+
|               name                | bs  |     inductor     |
+-----------------------------------+-----+------------------+
|            hf_T5_large            |  2  | pass_due_to_skip |
|   timm_vision_transformer_large   |  2  | pass_due_to_skip |
|           hf_GPT2_large           |  2  | pass_due_to_skip |
|           BERT_pytorch            |  2  |       pass       |
|        shufflenet_v2_x1_0         |  2  |       pass       |
|        mobilenet_v3_large         |  2  |       pass       |
|      nvidia_deeprecommender       |  2  |       pass       |
|   pytorch_CycleGAN_and_pix2pix    |  1  |       pass       |
|          pytorch_stargan          | 16  |       pass       |
|           pytorch_unet            |  2  |       pass       |
|             resnet18              |  2  |       pass       |
|             resnet50              |  2  |       pass       |
|      resnet50_quantized_qat       |  2  |       pass       |
|          resnext50_32x4d          |  2  |       pass       |
|         soft_actor_critic         | 256 |       pass       |
|           mobilenet_v2            |  2  |       pass       |
|        speech_transformer         |  2  |       pass       |
|           squeezenet1_1           |  2  |       pass       |
|         timm_efficientnet         |  2  |       pass       |
|            timm_nfnet             |  2  |       pass       |
|            timm_regnet            |  2  |       pass       |
|           timm_resnest            |  2  |       pass       |
|      timm_vision_transformer      |  2  |       pass       |
|            timm_vovnet            |  2  |       pass       |
|            tts_angular            |  2  |       pass       |
|               vgg16               |  2  |       pass       |
|    mobilenet_v2_quantized_qat     |  2  |       pass       |
|            mnasnet1_0             |  2  |       pass       |
|           fastNLP_Bert            |  2  |       pass       |
|       functorch_dp_cifar10        |  2  |       pass       |
|          LearningToPaint          |  2  |       pass       |
|            Super_SloMo            |  2  |       pass       |
|              alexnet              |  2  |       pass       |
| attention_is_all_you_need_pytorch |  2  |       pass       |
|               dcgan               |  2  |       pass       |
|              demucs               |  1  |       pass       |
|            densenet121            |  2  |       pass       |
|               dlrm                |  2  |       pass       |
|                drq                |  1  |       pass       |
|           lennard_jones           |  2  |       pass       |
|              yolov3               |  2  |       pass       |
|             hf_Albert             |  2  |       pass       |
|              hf_Bart              |  2  |       pass       |
|              hf_Bert              |  2  |       pass       |
|            hf_BigBird             |  2  |       pass       |
|           hf_DistilBert           |  2  |       pass       |
|              hf_GPT2              |  2  |       pass       |
|           hf_Longformer           |  2  |       pass       |
|            hf_Reformer            |  2  |       pass       |
|               hf_T5               |  2  |       pass       |
|            hf_T5_base             |  2  |       pass       |
|        Background_Matting         |  1  |  fail_accuracy   |
|     detectron2_fcos_r_50_fpn      |  0  |      0.0000      |
|          vision_maskrcnn          |  0  |      0.0000      |
+-----------------------------------+-----+------------------+

Compilation latency (sec)

+-----------------------------------+------+----------+
|               name                |  bs  | inductor |
+-----------------------------------+------+----------+
|            hf_T5_large            |  1   |  20.652  |
|          vision_maskrcnn          |  1   | 16.9233  |
|   timm_vision_transformer_large   |  8   | 12.9825  |
|           hf_GPT2_large           |  1   | 12.2959  |
|           hf_Longformer           |  1   | 11.5857  |
|              yolov3               |  8   | 11.1044  |
|            timm_nfnet             | 128  | 10.2945  |
|            hf_BigBird             |  1   | 10.2459  |
|            hf_T5_base             |  1   | 10.0108  |
|            densenet121            |  64  |  9.4044  |
|            Super_SloMo            |  6   |  6.6149  |
|            timm_regnet            |  32  |  5.5325  |
|        speech_transformer         |  1   |  5.0215  |
|              hf_Bart              |  1   |  4.9445  |
|           fastNLP_Bert            |  1   |  4.4597  |
| attention_is_all_you_need_pytorch |  32  |  4.2971  |
|               hf_T5               |  1   |  4.245   |
|           BERT_pytorch            |  2   |  4.2274  |
|         timm_efficientnet         |  64  |  4.1576  |
|              hf_Bert              |  1   |  3.9968  |
|      timm_vision_transformer      |  8   |  3.8982  |
|        mobilenet_v3_large         |  32  |  3.8922  |
|        shufflenet_v2_x1_0         |  64  |  3.7045  |
|           mobilenet_v2            |  16  |  3.6952  |
|            hf_Reformer            |  1   |  3.5863  |
|              hf_GPT2              |  1   |  3.5126  |
|          resnext50_32x4d          |  8   |  3.4358  |
|            mnasnet1_0             |  32  |  3.329   |
|        Background_Matting         |  1   |  3.2703  |
|            timm_vovnet            |  32  |  3.0503  |
|             hf_Albert             |  1   |  3.0439  |
|             resnet50              |  32  |  2.6625  |
|           timm_resnest            |  32  |  2.5714  |
|       functorch_dp_cifar10        |  64  |  2.4244  |
|           hf_DistilBert           |  1   |  2.098   |
|          LearningToPaint          |  96  |  1.7859  |
|           pytorch_unet            |  1   |  1.5454  |
|   pytorch_CycleGAN_and_pix2pix    |  1   |  1.5254  |
|          pytorch_stargan          |  16  |  1.3323  |
|             resnet18              |  8   |  1.2597  |
|              demucs               |  1   |  1.1289  |
|           squeezenet1_1           |  16  |  0.8397  |
|      nvidia_deeprecommender       | 256  |  0.7496  |
|               dlrm                | 2048 |  0.706   |
|               vgg16               |  4   |  0.5992  |
|            tts_angular            |  64  |  0.5919  |
|              alexnet              | 128  |  0.4548  |
|                drq                |  1   |  0.3724  |
|         soft_actor_critic         | 256  |  0.2193  |
|           lennard_jones           | 1000 |  0.1881  |
|               dcgan               | 256  |  0.1669  |
|    mobilenet_v2_quantized_qat     |  96  |  0.0845  |
|      resnet50_quantized_qat       |  32  |  0.0364  |
|     detectron2_fcos_r_50_fpn      |  0   |   nan    |
+-----------------------------------+------+----------+

Peak Memory Compression Ratio

+-----------------------------------+------+----------+
|               name                |  bs  | inductor |
+-----------------------------------+------+----------+
|              demucs               |  1   |  0.9989  |
|      resnet50_quantized_qat       |  32  |  0.9956  |
|               vgg16               |  4   |  0.9947  |
|            timm_regnet            |  32  |  0.9938  |
|   timm_vision_transformer_large   |  8   |  0.9936  |
|        Background_Matting         |  1   |  0.9933  |
|          LearningToPaint          |  96  |  0.9918  |
|              alexnet              | 128  |  0.9915  |
|            timm_nfnet             | 128  |  0.9913  |
|           pytorch_unet            |  1   |  0.9904  |
|            hf_BigBird             |  1   |  0.9901  |
|            timm_vovnet            |  32  |  0.9891  |
|            Super_SloMo            |  6   |  0.9878  |
|            densenet121            |  64  |  0.9866  |
|            mnasnet1_0             |  32  |  0.9851  |
|           lennard_jones           | 1000 |  0.9851  |
| attention_is_all_you_need_pytorch |  32  |  0.9834  |
|        mobilenet_v3_large         |  32  |  0.9826  |
|         soft_actor_critic         | 256  |  0.9822  |
|            tts_angular            |  64  |  0.9821  |
|        shufflenet_v2_x1_0         |  64  |  0.9818  |
|           mobilenet_v2            |  16  |  0.9808  |
|           hf_DistilBert           |  1   |  0.9805  |
|                drq                |  1   |  0.9801  |
|           hf_GPT2_large           |  1   |  0.9801  |
|              hf_Bart              |  1   |  0.9781  |
|           BERT_pytorch            |  2   |  0.9753  |
|            hf_T5_base             |  1   |  0.9752  |
|    mobilenet_v2_quantized_qat     |  96  |  0.9742  |
|         timm_efficientnet         |  64  |  0.9734  |
|             resnet50              |  32  |  0.9717  |
|          vision_maskrcnn          |  1   |  0.9709  |
|          pytorch_stargan          |  16  |  0.9709  |
|        speech_transformer         |  1   |  0.9708  |
|             resnet18              |  8   |   0.97   |
|           squeezenet1_1           |  16  |  0.9679  |
|   pytorch_CycleGAN_and_pix2pix    |  1   |  0.9673  |
|              hf_GPT2              |  1   |  0.964   |
|          resnext50_32x4d          |  8   |  0.9555  |
|             hf_Albert             |  1   |  0.9551  |
|              yolov3               |  8   |  0.9545  |
|               dlrm                | 2048 |  0.9473  |
|      timm_vision_transformer      |  8   |  0.9411  |
|           fastNLP_Bert            |  1   |  0.9254  |
|           hf_Longformer           |  1   |  0.9221  |
|       functorch_dp_cifar10        |  64  |  0.9091  |
|            hf_Reformer            |  1   |  0.822   |
|              hf_Bert              |  1   |  0.8084  |
|           timm_resnest            |  32  |  0.8031  |
|               dcgan               | 256  |  0.718   |
|            hf_T5_large            |  1   |  0.6768  |
|      nvidia_deeprecommender       | 256  |  0.587   |
|               hf_T5               |  1   |  0.5784  |
|     detectron2_fcos_r_50_fpn      |  0   |   nan    |
+-----------------------------------+------+----------+

huggingface suite with float32 precision

see more

Performance speedup

+-----------------------------------------+----+----------+
|                  name                   | bs | inductor |
+-----------------------------------------+----+----------+
|            XLNetLMHeadModel             | 4  |  7.481   |
|       MT5ForConditionalGeneration       | 2  |  6.932   |
|             XGLMForCausalLM             | 1  |  1.3679  |
|     M2M100ForConditionalGeneration      | 2  |  1.3395  |
|               DistillGPT2               | 1  |  1.3298  |
|             OPTForCausalLM              | 4  |  1.291   |
|           ElectraForCausalLM            | 1  |  1.2877  |
|          MobileBertForMaskedLM          | 16 |  1.2679  |
|               GoogleFnet                | 1  |  1.2672  |
|     MobileBertForQuestionAnswering      | 32 |  1.2178  |
|            YituTechConvBert             | 1  |  1.2023  |
|                 BigBird                 | 1  |  1.1067  |
|         MegatronBertForCausalLM         | 2  |  1.0607  |
|            AlbertForMaskedLM            | 2  |  1.0285  |
|                 T5Small                 | 1  |  1.0266  |
|       AlbertForQuestionAnswering        | 2  |  1.022   |
|          AllenaiLongformerBase          | 1  |  1.018   |
|     PegasusForConditionalGeneration     | 4  |  1.0109  |
|     PLBartForConditionalGeneration      | 8  |  0.9846  |
|            TrOCRForCausalLM             | 8  |  0.9816  |
|           RobertaForCausalLM            | 4  |  0.9761  |
|           PegasusForCausalLM            | 8  |  0.9758  |
|                CamemBert                | 1  |  0.9677  |
|           DebertaForMaskedLM            | 4  |  0.9663  |
|         Speech2Text2ForCausalLM         | 64 |  0.9482  |
|       DebertaForQuestionAnswering       | 4  |  0.9393  |
|      MBartForConditionalGeneration      | 8  |  0.9377  |
|            PLBartForCausalLM            | 16 |  0.9336  |
|      GPT2ForSequenceClassification      | 4  |  0.9188  |
|          DistilBertForMaskedLM          | 16 |  0.9062  |
|            MBartForCausalLM             | 16 |  0.8997  |
|    MegatronBertForQuestionAnswering     | 8  |  0.8808  |
|       RobertaForQuestionAnswering       | 64 |  0.8567  |
|    LayoutLMForSequenceClassification    | 16 |  0.854   |
|       T5ForConditionalGeneration        | 4  |  0.8515  |
|           LayoutLMForMaskedLM           | 16 |  0.8487  |
|             BartForCausalLM             | 2  |  0.8387  |
|     DistilBertForQuestionAnswering      | 32 |  0.8373  |
|        BertForQuestionAnswering         | 64 |  0.8344  |
|             BertForMaskedLM             | 64 |  0.8302  |
| BlenderbotSmallForConditionalGeneration | 32 |  0.8262  |
|      BartForConditionalGeneration       | 1  |  0.8234  |
|       ElectraForQuestionAnswering       | 64 |  0.7907  |
|       BlenderbotSmallForCausalLM        | 64 |  0.7829  |
+-----------------------------------------+----+----------+

Accuracy

+-----------------------------------------+----+----------+
|                  name                   | bs | inductor |
+-----------------------------------------+----+----------+
|            AlbertForMaskedLM            | 1  |   pass   |
|       AlbertForQuestionAnswering        | 1  |   pass   |
|      MBartForConditionalGeneration      | 1  |   pass   |
|       MT5ForConditionalGeneration       | 1  |   pass   |
|         MegatronBertForCausalLM         | 1  |   pass   |
|    MegatronBertForQuestionAnswering     | 1  |   pass   |
|          MobileBertForMaskedLM          | 1  |   pass   |
|     MobileBertForQuestionAnswering      | 1  |   pass   |
|             OPTForCausalLM              | 1  |   pass   |
|            PLBartForCausalLM            | 1  |   pass   |
|     PLBartForConditionalGeneration      | 1  |   pass   |
|           PegasusForCausalLM            | 1  |   pass   |
|     PegasusForConditionalGeneration     | 1  |   pass   |
|           RobertaForCausalLM            | 1  |   pass   |
|       RobertaForQuestionAnswering       | 1  |   pass   |
|         Speech2Text2ForCausalLM         | 1  |   pass   |
|       T5ForConditionalGeneration        | 1  |   pass   |
|                 T5Small                 | 1  |   pass   |
|            TrOCRForCausalLM             | 1  |   pass   |
|             XGLMForCausalLM             | 1  |   pass   |
|            XLNetLMHeadModel             | 1  |   pass   |
|            MBartForCausalLM             | 1  |   pass   |
|     M2M100ForConditionalGeneration      | 1  |   pass   |
|    LayoutLMForSequenceClassification    | 1  |   pass   |
|                CamemBert                | 1  |   pass   |
|          AllenaiLongformerBase          | 1  |   pass   |
|             BartForCausalLM             | 1  |   pass   |
|      BartForConditionalGeneration       | 1  |   pass   |
|             BertForMaskedLM             | 1  |   pass   |
|        BertForQuestionAnswering         | 1  |   pass   |
|                 BigBird                 | 1  |   pass   |
|       BlenderbotSmallForCausalLM        | 1  |   pass   |
| BlenderbotSmallForConditionalGeneration | 1  |   pass   |
|           DebertaForMaskedLM            | 1  |   pass   |
|           LayoutLMForMaskedLM           | 1  |   pass   |
|       DebertaForQuestionAnswering       | 1  |   pass   |
|          DistilBertForMaskedLM          | 1  |   pass   |
|     DistilBertForQuestionAnswering      | 1  |   pass   |
|               DistillGPT2               | 1  |   pass   |
|           ElectraForCausalLM            | 1  |   pass   |
|       ElectraForQuestionAnswering       | 1  |   pass   |
|      GPT2ForSequenceClassification      | 1  |   pass   |
|               GoogleFnet                | 1  |   pass   |
|            YituTechConvBert             | 1  |   pass   |
+-----------------------------------------+----+----------+

Compilation latency (sec)

+-----------------------------------------+----+----------+
|                  name                   | bs | inductor |
+-----------------------------------------+----+----------+
|     MobileBertForQuestionAnswering      | 32 | 14.3407  |
|          MobileBertForMaskedLM          | 16 | 14.3272  |
|          AllenaiLongformerBase          | 1  | 12.8257  |
|      MBartForConditionalGeneration      | 8  | 11.0707  |
|     PegasusForConditionalGeneration     | 4  | 10.7008  |
|                 BigBird                 | 1  | 10.3945  |
|       DebertaForQuestionAnswering       | 4  | 10.3599  |
|      BartForConditionalGeneration       | 1  | 10.1184  |
|           DebertaForMaskedLM            | 4  |  9.705   |
|    MegatronBertForQuestionAnswering     | 8  |  9.4832  |
|     M2M100ForConditionalGeneration      | 2  |  9.4558  |
|         MegatronBertForCausalLM         | 2  |  9.3521  |
|             XGLMForCausalLM             | 1  |  7.5338  |
| BlenderbotSmallForConditionalGeneration | 32 |  6.9944  |
|            YituTechConvBert             | 1  |  6.4043  |
|       ElectraForQuestionAnswering       | 64 |  6.0649  |
|       T5ForConditionalGeneration        | 4  |  5.9274  |
|        BertForQuestionAnswering         | 64 |  5.5988  |
|           LayoutLMForMaskedLM           | 16 |   5.4    |
|       MT5ForConditionalGeneration       | 2  |  5.2979  |
|     PLBartForConditionalGeneration      | 8  |  5.2489  |
|             BertForMaskedLM             | 64 |  5.2009  |
|            MBartForCausalLM             | 16 |  5.0805  |
|    LayoutLMForSequenceClassification    | 16 |  5.0601  |
|      GPT2ForSequenceClassification      | 4  |  4.9866  |
|           RobertaForCausalLM            | 4  |  4.6449  |
|       RobertaForQuestionAnswering       | 64 |  4.5597  |
|             BartForCausalLM             | 2  |  4.5368  |
|             OPTForCausalLM              | 4  |  4.379   |
|                 T5Small                 | 1  |  4.3357  |
|           PegasusForCausalLM            | 8  |  4.3134  |
|                CamemBert                | 1  |  4.1945  |
|            TrOCRForCausalLM             | 8  |  4.1408  |
|           ElectraForCausalLM            | 1  |  4.0948  |
|            AlbertForMaskedLM            | 2  |  3.5141  |
|       AlbertForQuestionAnswering        | 2  |  3.5065  |
|       BlenderbotSmallForCausalLM        | 64 |  3.2783  |
|     DistilBertForQuestionAnswering      | 32 |  2.9662  |
|               GoogleFnet                | 1  |  2.6108  |
|          DistilBertForMaskedLM          | 16 |  2.4159  |
|               DistillGPT2               | 1  |  2.3806  |
|            PLBartForCausalLM            | 16 |  2.2758  |
|         Speech2Text2ForCausalLM         | 64 |  2.1607  |
|            XLNetLMHeadModel             | 4  | -14.658  |
+-----------------------------------------+----+----------+

Peak Memory Compression Ratio

+-----------------------------------------+----+----------+
|                  name                   | bs | inductor |
+-----------------------------------------+----+----------+
|            AlbertForMaskedLM            | 2  |  0.9964  |
|       AlbertForQuestionAnswering        | 2  |  0.9962  |
|             BertForMaskedLM             | 64 |  0.9954  |
|       BlenderbotSmallForCausalLM        | 64 |  0.9954  |
|      GPT2ForSequenceClassification      | 4  |  0.9954  |
|       ElectraForQuestionAnswering       | 64 |  0.9953  |
|       T5ForConditionalGeneration        | 4  |  0.9939  |
|             BartForCausalLM             | 2  |  0.9936  |
|           DebertaForMaskedLM            | 4  |  0.9922  |
|                 BigBird                 | 1  |  0.9914  |
|            MBartForCausalLM             | 16 |  0.9912  |
|       DebertaForQuestionAnswering       | 4  |  0.9908  |
|      BartForConditionalGeneration       | 1  |  0.9906  |
|            PLBartForCausalLM            | 16 |  0.9902  |
| BlenderbotSmallForConditionalGeneration | 32 |  0.9897  |
|            TrOCRForCausalLM             | 8  |  0.9896  |
|           PegasusForCausalLM            | 8  |  0.9894  |
|               GoogleFnet                | 1  |  0.9891  |
|     DistilBertForQuestionAnswering      | 32 |  0.9889  |
|     PegasusForConditionalGeneration     | 4  |  0.9886  |
|         Speech2Text2ForCausalLM         | 64 |  0.9885  |
|           LayoutLMForMaskedLM           | 16 |  0.988   |
|          DistilBertForMaskedLM          | 16 |  0.9874  |
|      MBartForConditionalGeneration      | 8  |  0.9856  |
|    LayoutLMForSequenceClassification    | 16 |  0.9853  |
|        BertForQuestionAnswering         | 64 |  0.9826  |
|     PLBartForConditionalGeneration      | 8  |  0.9815  |
|             OPTForCausalLM              | 4  |  0.9807  |
|               DistillGPT2               | 1  |  0.9755  |
|          AllenaiLongformerBase          | 1  |  0.974   |
|     M2M100ForConditionalGeneration      | 2  |  0.9724  |
|          MobileBertForMaskedLM          | 16 |  0.9715  |
|           RobertaForCausalLM            | 4  |  0.9678  |
|             XGLMForCausalLM             | 1  |  0.9657  |
|       RobertaForQuestionAnswering       | 64 |  0.9552  |
|           ElectraForCausalLM            | 1  |  0.9379  |
|            XLNetLMHeadModel             | 4  |  0.9116  |
|                CamemBert                | 1  |  0.8447  |
|                 T5Small                 | 1  |  0.8396  |
|     MobileBertForQuestionAnswering      | 32 |  0.8228  |
|            YituTechConvBert             | 1  |  0.8046  |
|         MegatronBertForCausalLM         | 2  |  0.6934  |
|    MegatronBertForQuestionAnswering     | 8  |  0.6877  |
|       MT5ForConditionalGeneration       | 2  |  0.5174  |
+-----------------------------------------+----+----------+

timm_models suite with float32 precision

see more

Performance speedup

+---------------------------------+-----+----------+
|              name               | bs  | inductor |
+---------------------------------+-----+----------+
|        res2net101_26w_4s        | 64  |  1.286   |
|           mnasnet_100           | 128 |  1.2825  |
|          spnasnet_100           | 128 |  1.2711  |
|          inception_v3           | 128 |  1.2154  |
|           fbnetc_100            | 128 |  1.2065  |
|       gluon_inception_v3        | 128 |  1.1991  |
|        adv_inception_v3         | 128 |  1.1929  |
|          ghostnet_100           | 128 |  1.1589  |
|        convmixer_768_32         | 32  |  1.1486  |
|      mobilenetv3_large_100      | 128 |  1.1401  |
|            gernet_l             | 128 |  1.1394  |
|             dpn107              | 32  |  1.1393  |
|         mobilenetv2_100         | 128 |  1.1004  |
|           volo_d1_224           | 64  |  1.0784  |
|        res2net50_14w_8s         |  2  |  1.0773  |
|            repvgg_a2            | 128 |  1.072   |
|            hrnet_w18            |  2  |  1.0713  |
|            fbnetv3_b            | 128 |  1.0666  |
|           resnest101e           | 32  |  1.0534  |
|           selecsls42b           | 128 |  1.0523  |
|           regnety_002           | 128 |  1.0394  |
|            lcnet_050            | 128 |  1.0392  |
|     swsl_resnext101_32x16d      | 32  |  1.0289  |
|          pnasnet5large          | 16  |  1.0235  |
|             dla102              | 64  |  0.9799  |
|           tf_mixnet_l           | 64  |  0.9699  |
|          cait_m36_384           |  2  |  0.9604  |
|         poolformer_m36          | 64  |  0.9602  |
|        ese_vovnet19b_dw         | 128 |  0.9537  |
|          gmixer_24_224          | 64  |  0.9481  |
|      xcit_large_24_p8_224       |  5  |  0.9399  |
|            mixnet_l             | 64  |  0.9373  |
|  swin_base_patch4_window7_224   | 64  |  0.8761  |
|      beit_base_patch16_224      | 64  |  0.8741  |
|            nfnet_l0             | 64  |  0.8696  |
|           dm_nfnet_f0           | 128 |  0.859   |
| deit_base_distilled_patch16_224 | 64  |  0.8468  |
|           convit_base           | 32  |  0.843   |
|      vit_base_patch16_224       | 64  |  0.8349  |
|          resmlp_12_224          | 128 |  0.8168  |
|          jx_nest_base           | 32  |  0.8133  |
|        tnt_s_patch16_224        | 64  |  0.8119  |
|           rexnet_100            | 128 |  0.8068  |
|         visformer_small         | 128 |  0.8049  |
|           res2next50            |  2  |  0.8005  |
|          mixer_b16_224          | 64  |  0.7952  |
|          convnext_base          | 32  |  0.7919  |
|            pit_b_224            | 64  |  0.7642  |
|            tinynet_a            | 128 |  0.7436  |
|       tf_efficientnet_b0        | 128 |  0.7389  |
|         coat_lite_mini          | 128 |  0.7325  |
|        gluon_xception65         | 32  |  0.7183  |
|        twins_pcpvt_base         | 32  |  0.7145  |
|         crossvit_9_240          | 64  |  0.6909  |
|          cspdarknet53           | 64  |  0.6884  |
|          gmlp_s16_224           | 64  |  0.5267  |
|           mobilevit_s           | 32  |  0.502   |
|       eca_botnext26ts_256       |  0  |   0.0    |
|        eca_halonext26ts         |  0  |   0.0    |
|          botnet26t_256          |  0  |   0.0    |
|        sebotnet33ts_256         |  0  |   0.0    |
+---------------------------------+-----+----------+

Accuracy

+---------------------------------+----+---------------+
|              name               | bs |   inductor    |
+---------------------------------+----+---------------+
|        adv_inception_v3         | 2  |     pass      |
|      beit_base_patch16_224      | 2  |     pass      |
|         mobilenetv2_100         | 2  |     pass      |
|      mobilenetv3_large_100      | 2  |     pass      |
|           mobilevit_s           | 2  |     pass      |
|            nfnet_l0             | 2  |     pass      |
|            pit_b_224            | 2  |     pass      |
|          pnasnet5large          | 2  |     pass      |
|         poolformer_m36          | 2  |     pass      |
|           regnety_002           | 2  |     pass      |
|            repvgg_a2            | 2  |     pass      |
|        res2net101_26w_4s        | 2  |     pass      |
|        res2net50_14w_8s         | 2  |     pass      |
|           res2next50            | 2  |     pass      |
|          resmlp_12_224          | 2  |     pass      |
|           resnest101e           | 2  |     pass      |
|           rexnet_100            | 2  |     pass      |
|           selecsls42b           | 2  |     pass      |
|          spnasnet_100           | 2  |     pass      |
|  swin_base_patch4_window7_224   | 2  |     pass      |
|     swsl_resnext101_32x16d      | 2  |     pass      |
|       tf_efficientnet_b0        | 2  |     pass      |
|           tf_mixnet_l           | 2  |     pass      |
|            tinynet_a            | 2  |     pass      |
|        tnt_s_patch16_224        | 2  |     pass      |
|        twins_pcpvt_base         | 2  |     pass      |
|         visformer_small         | 2  |     pass      |
|      vit_base_patch16_224       | 2  |     pass      |
|           volo_d1_224           | 2  |     pass      |
|           mnasnet_100           | 2  |     pass      |
|            mixnet_l             | 2  |     pass      |
|          mixer_b16_224          | 2  |     pass      |
|        ese_vovnet19b_dw         | 2  |     pass      |
|         coat_lite_mini          | 2  |     pass      |
|           convit_base           | 2  |     pass      |
|        convmixer_768_32         | 2  |     pass      |
|          convnext_base          | 2  |     pass      |
|         crossvit_9_240          | 2  |     pass      |
|          cspdarknet53           | 2  |     pass      |
| deit_base_distilled_patch16_224 | 2  |     pass      |
|             dla102              | 2  |     pass      |
|           dm_nfnet_f0           | 2  |     pass      |
|             dpn107              | 2  |     pass      |
|            lcnet_050            | 2  |     pass      |
|      xcit_large_24_p8_224       | 2  |     pass      |
|           fbnetc_100            | 2  |     pass      |
|            fbnetv3_b            | 2  |     pass      |
|            gernet_l             | 2  |     pass      |
|       gluon_inception_v3        | 2  |     pass      |
|        gluon_xception65         | 2  |     pass      |
|          gmixer_24_224          | 2  |     pass      |
|          gmlp_s16_224           | 2  |     pass      |
|            hrnet_w18            | 2  |     pass      |
|          inception_v3           | 2  |     pass      |
|          jx_nest_base           | 2  |     pass      |
|        eca_halonext26ts         | 2  |  fail_to_run  |
|        sebotnet33ts_256         | 2  |  fail_to_run  |
|          botnet26t_256          | 2  |  fail_to_run  |
|       eca_botnext26ts_256       | 2  |  fail_to_run  |
|          ghostnet_100           | 2  | fail_accuracy |
|          cait_m36_384           | 2  | fail_accuracy |
+---------------------------------+----+---------------+

Compilation latency (sec)

+---------------------------------+-----+----------+
|              name               | bs  | inductor |
+---------------------------------+-----+----------+
|          pnasnet5large          | 16  | 19.2248  |
|            hrnet_w18            |  2  | 18.9544  |
|  swin_base_patch4_window7_224   | 64  | 14.8893  |
|      xcit_large_24_p8_224       |  5  | 14.0629  |
|           mobilevit_s           | 32  | 13.4383  |
|          cait_m36_384           |  2  | 13.0711  |
|         poolformer_m36          | 64  |  12.376  |
|        twins_pcpvt_base         | 32  | 11.2295  |
|           dm_nfnet_f0           | 128 | 10.6087  |
|           resnest101e           | 32  | 10.3014  |
|           tf_mixnet_l           | 64  |  9.6655  |
|        res2net101_26w_4s        | 64  |  9.6545  |
|          jx_nest_base           | 32  |  9.5788  |
|        tnt_s_patch16_224        | 64  |  9.3977  |
|            mixnet_l             | 64  |  9.3847  |
|        gluon_xception65         | 32  |  9.3583  |
|        res2net50_14w_8s         |  2  |  8.9778  |
|          gmlp_s16_224           | 64  |  8.6621  |
|             dpn107              | 32  |  8.6491  |
|            fbnetv3_b            | 128 |  8.4694  |
|           volo_d1_224           | 64  |  8.0304  |
|          convnext_base          | 32  |  7.4253  |
|         crossvit_9_240          | 64  |  7.0807  |
|            nfnet_l0             | 64  |  7.0103  |
|            pit_b_224            | 64  |  6.5919  |
|             dla102              | 64  |  6.5173  |
|          cspdarknet53           | 64  |  6.4078  |
|         coat_lite_mini          | 128 |  6.2778  |
|       tf_efficientnet_b0        | 128 |  6.0339  |
|     swsl_resnext101_32x16d      | 32  |  5.8507  |
|          ghostnet_100           | 128 |   5.82   |
|           convit_base           | 32  |  5.7277  |
|            tinynet_a            | 128 |  5.679   |
|           rexnet_100            | 128 |  5.6751  |
|          gmixer_24_224          | 64  |  5.6566  |
|           res2next50            |  2  |  5.3908  |
|      vit_base_patch16_224       | 64  |  5.0519  |
|         visformer_small         | 128 |  4.8499  |
|       gluon_inception_v3        | 128 |   4.78   |
|          mixer_b16_224          | 64  |  4.7777  |
|           fbnetc_100            | 128 |  4.7003  |
|      beit_base_patch16_224      | 64  |  4.6882  |
|          inception_v3           | 128 |  4.5974  |
|         mobilenetv2_100         | 128 |  4.5727  |
|        adv_inception_v3         | 128 |  4.5165  |
|            gernet_l             | 128 |  4.3313  |
| deit_base_distilled_patch16_224 | 64  |  4.3017  |
|           regnety_002           | 128 |  4.1075  |
|            repvgg_a2            | 128 |  4.0545  |
|          spnasnet_100           | 128 |  3.9151  |
|      mobilenetv3_large_100      | 128 |  3.5092  |
|           mnasnet_100           | 128 |  3.2211  |
|        convmixer_768_32         | 32  |  3.063   |
|        ese_vovnet19b_dw         | 128 |  2.7839  |
|            lcnet_050            | 128 |  2.1599  |
|           selecsls42b           | 128 |  2.1297  |
|          resmlp_12_224          | 128 |  2.0739  |
|          botnet26t_256          |  0  |   nan    |
|       eca_botnext26ts_256       |  0  |   nan    |
|        eca_halonext26ts         |  0  |   nan    |
|        sebotnet33ts_256         |  0  |   nan    |
+---------------------------------+-----+----------+

Peak Memory Compression Ratio

+---------------------------------+-----+----------+
|              name               | bs  | inductor |
+---------------------------------+-----+----------+
|        ese_vovnet19b_dw         | 128 |  0.9973  |
|     swsl_resnext101_32x16d      | 32  |  0.9967  |
|      vit_base_patch16_224       | 64  |  0.9965  |
|           dm_nfnet_f0           | 128 |  0.9965  |
|         visformer_small         | 128 |  0.9964  |
|      beit_base_patch16_224      | 64  |  0.9962  |
|          mixer_b16_224          | 64  |  0.9958  |
|            gernet_l             | 128 |  0.9956  |
|         coat_lite_mini          | 128 |  0.9955  |
|        gluon_xception65         | 32  |  0.9955  |
|          cspdarknet53           | 64  |  0.9955  |
|            pit_b_224            | 64  |  0.9954  |
|            fbnetv3_b            | 128 |  0.995   |
|           convit_base           | 32  |  0.9949  |
|           fbnetc_100            | 128 |  0.9947  |
|       tf_efficientnet_b0        | 128 |  0.9944  |
|           rexnet_100            | 128 |  0.9943  |
|          spnasnet_100           | 128 |  0.9943  |
|          gmixer_24_224          | 64  |  0.9943  |
|  swin_base_patch4_window7_224   | 64  |  0.9938  |
|         mobilenetv2_100         | 128 |  0.9938  |
|            repvgg_a2            | 128 |  0.9935  |
|          jx_nest_base           | 32  |  0.9933  |
|            nfnet_l0             | 64  |  0.9925  |
|          convnext_base          | 32  |  0.9922  |
|           mobilevit_s           | 32  |  0.9921  |
|           tf_mixnet_l           | 64  |  0.992   |
|         poolformer_m36          | 64  |  0.9916  |
|             dpn107              | 32  |  0.9916  |
|            mixnet_l             | 64  |  0.9916  |
|          ghostnet_100           | 128 |  0.9914  |
|          pnasnet5large          | 16  |  0.9914  |
|           selecsls42b           | 128 |  0.9907  |
|           mnasnet_100           | 128 |  0.9907  |
|           volo_d1_224           | 64  |  0.9898  |
|      xcit_large_24_p8_224       |  5  |  0.9887  |
|           regnety_002           | 128 |  0.9886  |
|          cait_m36_384           |  2  |  0.9885  |
|       gluon_inception_v3        | 128 |  0.985   |
|        adv_inception_v3         | 128 |  0.9849  |
|          inception_v3           | 128 |  0.9845  |
|          resmlp_12_224          | 128 |  0.9842  |
|      mobilenetv3_large_100      | 128 |  0.9839  |
| deit_base_distilled_patch16_224 | 64  |  0.9824  |
|            tinynet_a            | 128 |  0.9789  |
|            lcnet_050            | 128 |  0.9726  |
|             dla102              | 64  |  0.9696  |
|        convmixer_768_32         | 32  |  0.9682  |
|        res2net101_26w_4s        | 64  |  0.9647  |
|           res2next50            |  2  |  0.9514  |
|           resnest101e           | 32  |  0.951   |
|         crossvit_9_240          | 64  |  0.9487  |
|        twins_pcpvt_base         | 32  |  0.9332  |
|        res2net50_14w_8s         |  2  |  0.9328  |
|          gmlp_s16_224           | 64  |  0.9267  |
|            hrnet_w18            |  2  |  0.8875  |
|        tnt_s_patch16_224        | 64  |  0.7729  |
|          botnet26t_256          |  0  |   nan    |
|       eca_botnext26ts_256       |  0  |   nan    |
|        eca_halonext26ts         |  0  |   nan    |
|        sebotnet33ts_256         |  0  |   nan    |
+---------------------------------+-----+----------+

[yudongsi]

Performance Dashboard for float32 precision -- Single-Socket Multi-threads

Executive Summary

see more

We evaluate different backends across three benchmark suites - torchbench, huggingface and timm. We run these experiments on Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz. Each experiment runs one iteration of forward pass. For accuracy, we check the numerical correctness of forward pass outputs by comparing with native pytorch. We measure speedup by normalizing against the performance of native pytorch. We report mean compilation latency numbers and peak memory footprint reduction ratio.

Caveats

1. Batch size has been reduced to workaround OOM errors. Work is in progress to reduce peak memory footprint.
2. Experiments do not cover dynamic shapes.
3. Experimental setup does not have optimizer.

To measure performance, compilation latency and memory footprint reduction, we remove the models that fail accuracy checks.

Passrate

+----------+------------+-------------+-------------+
| Compiler | torchbench | huggingface | timm_models |
+----------+------------+-------------+-------------+
| inductor | 89%, 49/55 | 100%, 44/44 | 89%, 54/61  |
+----------+------------+-------------+-------------+

Geometric mean speedup

+----------+------------+-------------+-------------+
| Compiler | torchbench | huggingface | timm_models |
+----------+------------+-------------+-------------+
| inductor |   1.06x    |    1.07x    |    1.08x    |
+----------+------------+-------------+-------------+

Mean compilation time (seconds)

+----------+------------+-------------+-------------+
| Compiler | torchbench | huggingface | timm_models |
+----------+------------+-------------+-------------+
| inductor |   10.95    |    14.59    |    16.49    |
+----------+------------+-------------+-------------+

Peak memory footprint compression ratio (higher is better)

+----------+------------+-------------+-------------+
| Compiler | torchbench | huggingface | timm_models |
+----------+------------+-------------+-------------+
| inductor |   0.93x    |    0.97x    |    0.99x    |
+----------+------------+-------------+-------------+

torchbench suite with float32 precision

see more

Performance speedup

+-----------------------------------+------+----------+
|               name                |  bs  | inductor |
+-----------------------------------+------+----------+
|           timm_resnest            |  32  |  1.2876  |
|            densenet121            |  64  |  1.2032  |
|        shufflenet_v2_x1_0         |  64  |  1.1822  |
|             resnet18              |  8   |  1.1751  |
|       functorch_dp_cifar10        |  64  |  1.1673  |
|              alexnet              | 128  |   1.15   |
|           squeezenet1_1           |  16  |  1.1478  |
|            mnasnet1_0             |  32  |  1.1333  |
|           pytorch_unet            |  1   |  1.1325  |
|        mobilenet_v3_large         |  32  |  1.1291  |
|            timm_vovnet            |  32  |  1.1207  |
|          resnext50_32x4d          |  8   |  1.1188  |
|   pytorch_CycleGAN_and_pix2pix    |  1   |  1.1158  |
|             resnet50              |  32  |  1.1079  |
|           mobilenet_v2            |  16  |  1.0973  |
|          vision_maskrcnn          |  1   |  1.0919  |
|               vgg16               |  4   |  1.087   |
|                drq                |  1   |  1.0844  |
|               dcgan               | 256  |  1.0739  |
|            hf_T5_large            |  1   |  1.0637  |
|            timm_regnet            |  32  |  1.0604  |
|               dlrm                | 2048 |  1.0598  |
|          LearningToPaint          |  96  |  1.0565  |
|              yolov3               |  8   |  1.0482  |
|            Super_SloMo            |  6   |  1.0463  |
|            hf_BigBird             |  1   |  1.0328  |
|          pytorch_stargan          |  16  |  1.0299  |
|               hf_T5               |  1   |  1.0255  |
|        Background_Matting         |  1   |  1.0247  |
|     detectron2_fcos_r_50_fpn      |  1   |  1.0231  |
|            hf_Reformer            |  1   |  1.0204  |
|              demucs               |  1   |  1.0015  |
|            tts_angular            |  64  |  1.001   |
|    mobilenet_v2_quantized_qat     |  96  |  1.0008  |
|      resnet50_quantized_qat       |  32  |  0.9899  |
|           BERT_pytorch            |  2   |  0.9855  |
|           hf_GPT2_large           |  1   |  0.9577  |
|              hf_GPT2              |  1   |  0.9225  |
|           hf_Longformer           |  1   |  0.9222  |
|             hf_Albert             |  1   |  0.8953  |
|            hf_T5_base             |  1   |  0.8771  |
|   timm_vision_transformer_large   |  8   |  0.8726  |
|      nvidia_deeprecommender       | 256  |  0.8412  |
|           hf_DistilBert           |  1   |  0.8293  |
|            timm_nfnet             | 128  |  0.8142  |
|         timm_efficientnet         |  64  |  0.8089  |
|              hf_Bart              |  1   |  0.7965  |
|              hf_Bert              |  1   |  0.796   |
|      timm_vision_transformer      |  8   |  0.7337  |
| attention_is_all_you_need_pytorch |  32  |  0.7245  |
|           lennard_jones           | 1000 |  0.3547  |
|        speech_transformer         |  0   |   0.0    |
|         soft_actor_critic         |  0   |   0.0    |
|             tacotron2             |  0   |   0.0    |
|           fastNLP_Bert            |  0   |   0.0    |
+-----------------------------------+------+----------+

Accuracy

+-----------------------------------+----+------------------+
|               name                | bs |     inductor     |
+-----------------------------------+----+------------------+
|            hf_T5_large            | 2  | pass_due_to_skip |
|           hf_GPT2_large           | 2  | pass_due_to_skip |
|   timm_vision_transformer_large   | 2  | pass_due_to_skip |
|               hf_T5               | 2  |       pass       |
|            hf_Reformer            | 2  |       pass       |
|            hf_T5_base             | 2  |       pass       |
|          LearningToPaint          | 2  |       pass       |
|            Super_SloMo            | 2  |       pass       |
|              alexnet              | 2  |       pass       |
| attention_is_all_you_need_pytorch | 2  |       pass       |
|               dcgan               | 2  |       pass       |
|              demucs               | 1  |       pass       |
|            densenet121            | 2  |       pass       |
|               dlrm                | 2  |       pass       |
|                drq                | 1  |       pass       |
|              yolov3               | 2  |       pass       |
|            mnasnet1_0             | 2  |       pass       |
|             hf_Albert             | 2  |       pass       |
|              hf_Bart              | 2  |       pass       |
|              hf_Bert              | 2  |       pass       |
|            hf_BigBird             | 2  |       pass       |
|           hf_DistilBert           | 2  |       pass       |
|              hf_GPT2              | 2  |       pass       |
|           hf_Longformer           | 2  |       pass       |
|       functorch_dp_cifar10        | 2  |       pass       |
|           lennard_jones           | 2  |       pass       |
|          resnext50_32x4d          | 2  |       pass       |
|           BERT_pytorch            | 2  |       pass       |
|      resnet50_quantized_qat       | 2  |       pass       |
|           mobilenet_v2            | 2  |       pass       |
|    mobilenet_v2_quantized_qat     | 2  |       pass       |
|        mobilenet_v3_large         | 2  |       pass       |
|      nvidia_deeprecommender       | 2  |       pass       |
|   pytorch_CycleGAN_and_pix2pix    | 1  |       pass       |
|          pytorch_stargan          | 16 |       pass       |
|           pytorch_unet            | 2  |       pass       |
|               vgg16               | 2  |       pass       |
|             resnet50              | 2  |       pass       |
|             resnet18              | 2  |       pass       |
|        Background_Matting         | 1  |       pass       |
|        shufflenet_v2_x1_0         | 2  |       pass       |
|           squeezenet1_1           | 2  |       pass       |
|         timm_efficientnet         | 2  |       pass       |
|            timm_nfnet             | 2  |       pass       |
|            timm_regnet            | 2  |       pass       |
|           timm_resnest            | 2  |       pass       |
|      timm_vision_transformer      | 2  |       pass       |
|            timm_vovnet            | 2  |       pass       |
|            tts_angular            | 2  |       pass       |
|     detectron2_fcos_r_50_fpn      | 2  |   fail_to_run    |
|           fastNLP_Bert            | 0  |      0.0000      |
|         soft_actor_critic         | 0  |      0.0000      |
|             tacotron2             | 0  |      0.0000      |
|          vision_maskrcnn          | 0  |      0.0000      |
|        speech_transformer         | 0  |      0.0000      |
+-----------------------------------+----+------------------+

Compilation latency (sec)

+-----------------------------------+------+----------+
|               name                |  bs  | inductor |
+-----------------------------------+------+----------+
|     detectron2_fcos_r_50_fpn      |  1   | 60.6662  |
|          vision_maskrcnn          |  1   | 57.9893  |
|            hf_T5_large            |  1   | 30.2159  |
|            hf_BigBird             |  1   | 20.3531  |
|           hf_Longformer           |  1   | 19.4456  |
|   timm_vision_transformer_large   |  8   | 19.3107  |
|           BERT_pytorch            |  2   | 19.2475  |
|            timm_nfnet             | 128  | 19.1044  |
|           hf_GPT2_large           |  1   | 19.0724  |
|              yolov3               |  8   | 18.4934  |
|            hf_T5_base             |  1   |  18.095  |
|            densenet121            |  64  | 16.2855  |
|            Super_SloMo            |  6   |  14.675  |
|         timm_efficientnet         |  64  | 13.0065  |
|              hf_Bart              |  1   | 12.0408  |
|            timm_regnet            |  32  | 11.9215  |
|            hf_Reformer            |  1   | 11.5293  |
| attention_is_all_you_need_pytorch |  32  |  11.421  |
|               hf_T5               |  1   | 11.3392  |
|              hf_Bert              |  1   | 10.6351  |
|        mobilenet_v3_large         |  32  | 10.5692  |
|        shufflenet_v2_x1_0         |  64  | 10.4591  |
|        Background_Matting         |  1   |  10.313  |
|              hf_GPT2              |  1   | 10.1996  |
|           pytorch_unet            |  1   | 10.1988  |
|      timm_vision_transformer      |  8   | 10.1087  |
|           mobilenet_v2            |  16  | 10.0596  |
|          resnext50_32x4d          |  8   |  9.8076  |
|            mnasnet1_0             |  32  |  9.7355  |
|             hf_Albert             |  1   |  9.6999  |
|             resnet50              |  32  |  9.6626  |
|            timm_vovnet            |  32  |  9.639   |
|           timm_resnest            |  32  |  8.6942  |
|           hf_DistilBert           |  1   |  8.6865  |
|       functorch_dp_cifar10        |  64  |  8.1404  |
|   pytorch_CycleGAN_and_pix2pix    |  1   |  7.8394  |
|          pytorch_stargan          |  16  |  7.7223  |
|          LearningToPaint          |  96  |  7.7173  |
|             resnet18              |  8   |  7.4809  |
|           squeezenet1_1           |  16  |  7.2604  |
|               vgg16               |  4   |  7.1666  |
|               dlrm                | 2048 |  7.0587  |
|            tts_angular            |  64  |  7.012   |
|      nvidia_deeprecommender       | 256  |  6.9814  |
|              alexnet              | 128  |  6.8342  |
|                drq                |  1   |  6.5838  |
|               dcgan               | 256  |  6.5091  |
|           lennard_jones           | 1000 |  6.4467  |
|              demucs               |  1   |  1.4501  |
|      resnet50_quantized_qat       |  32  |  0.1674  |
|    mobilenet_v2_quantized_qat     |  96  |  0.1381  |
|           fastNLP_Bert            |  0   |   nan    |
|         soft_actor_critic         |  0   |   nan    |
|        speech_transformer         |  0   |   nan    |
|             tacotron2             |  0   |   nan    |
+-----------------------------------+------+----------+

Peak Memory Compression Ratio

+-----------------------------------+------+----------+
|               name                |  bs  | inductor |
+-----------------------------------+------+----------+
|              demucs               |  1   |  0.9988  |
|      resnet50_quantized_qat       |  32  |  0.9962  |
|            hf_T5_base             |  1   |  0.9954  |
|            timm_nfnet             | 128  |  0.9928  |
|        Background_Matting         |  1   |  0.9925  |
|               vgg16               |  4   |  0.9925  |
|          LearningToPaint          |  96  |  0.9903  |
|            Super_SloMo            |  6   |  0.9903  |
|         timm_efficientnet         |  64  |  0.9902  |
|            hf_BigBird             |  1   |  0.9899  |
|           pytorch_unet            |  1   |  0.9891  |
|   timm_vision_transformer_large   |  8   |  0.9872  |
|            mnasnet1_0             |  32  |  0.9841  |
|           lennard_jones           | 1000 |  0.9824  |
| attention_is_all_you_need_pytorch |  32  |  0.982   |
|           mobilenet_v2            |  16  |  0.9801  |
|           hf_GPT2_large           |  1   |  0.9796  |
|        shufflenet_v2_x1_0         |  64  |  0.9793  |
|           hf_DistilBert           |  1   |  0.9792  |
|              alexnet              | 128  |  0.9791  |
|        mobilenet_v3_large         |  32  |  0.979   |
|          resnext50_32x4d          |  8   |  0.9787  |
|            timm_vovnet            |  32  |  0.9786  |
|     detectron2_fcos_r_50_fpn      |  1   |  0.9784  |
|                drq                |  1   |  0.9769  |
|              hf_Bart              |  1   |  0.9763  |
|    mobilenet_v2_quantized_qat     |  96  |  0.9732  |
|           BERT_pytorch            |  2   |  0.9725  |
|           timm_resnest            |  32  |  0.9704  |
|             resnet50              |  32  |  0.9696  |
|          pytorch_stargan          |  16  |  0.9637  |
|              yolov3               |  8   |  0.9567  |
|   pytorch_CycleGAN_and_pix2pix    |  1   |  0.9535  |
|              hf_GPT2              |  1   |  0.9527  |
|            tts_angular            |  64  |  0.9512  |
|               dlrm                | 2048 |  0.9468  |
|             hf_Albert             |  1   |  0.941   |
|      timm_vision_transformer      |  8   |  0.9341  |
|           hf_Longformer           |  1   |  0.9206  |
|       functorch_dp_cifar10        |  64  |  0.8988  |
|             resnet18              |  8   |  0.8541  |
|          vision_maskrcnn          |  1   |  0.853   |
|            hf_Reformer            |  1   |  0.8337  |
|            timm_regnet            |  32  |  0.8202  |
|           squeezenet1_1           |  16  |  0.7868  |
|            densenet121            |  64  |  0.7861  |
|              hf_Bert              |  1   |  0.7728  |
|               hf_T5               |  1   |  0.7683  |
|               dcgan               | 256  |  0.6899  |
|            hf_T5_large            |  1   |  0.6732  |
|      nvidia_deeprecommender       | 256  |  0.583   |
|           fastNLP_Bert            |  0   |   nan    |
|         soft_actor_critic         |  0   |   nan    |
|        speech_transformer         |  0   |   nan    |
|             tacotron2             |  0   |   nan    |
+-----------------------------------+------+----------+

huggingface suite with float32 precision

see more

Performance speedup

+-----------------------------------------+-----+----------+
|                  name                   | bs  | inductor |
+-----------------------------------------+-----+----------+
|       MT5ForConditionalGeneration       |  8  |  2.5426  |
|            XLNetLMHeadModel             | 32  |  1.9834  |
|     MobileBertForQuestionAnswering      | 64  |  1.3245  |
|               DistillGPT2               |  1  |  1.3185  |
|               GoogleFnet                |  1  |  1.2467  |
|            YituTechConvBert             |  1  |  1.1918  |
|          MobileBertForMaskedLM          | 32  |  1.1428  |
|                 BigBird                 |  1  |  1.0946  |
|     M2M100ForConditionalGeneration      |  8  |  1.0897  |
|             XGLMForCausalLM             |  8  |  1.0627  |
|          AllenaiLongformerBase          |  1  |  1.032   |
|                 T5Small                 |  1  |  1.0308  |
|            AlbertForMaskedLM            |  4  |  1.0299  |
|       AlbertForQuestionAnswering        |  4  |  1.0296  |
|             OPTForCausalLM              | 32  |  1.0154  |
|           DebertaForMaskedLM            |  4  |  0.9783  |
|      GPT2ForSequenceClassification      |  4  |  0.9597  |
|         Speech2Text2ForCausalLM         | 128 |  0.9463  |
|                CamemBert                |  1  |  0.9439  |
|       DebertaForQuestionAnswering       |  8  |  0.9296  |
|           RobertaForCausalLM            | 64  |  0.9035  |
|           ElectraForCausalLM            | 32  |  0.9019  |
|     PLBartForConditionalGeneration      | 16  |  0.8948  |
|      MBartForConditionalGeneration      | 16  |  0.8879  |
|            PLBartForCausalLM            | 32  |  0.8864  |
|    LayoutLMForSequenceClassification    | 16  |  0.8858  |
|         MegatronBertForCausalLM         | 16  |  0.8855  |
|     PegasusForConditionalGeneration     | 16  |  0.8843  |
|       T5ForConditionalGeneration        |  4  |  0.878   |
|            TrOCRForCausalLM             | 32  |  0.8737  |
|       RobertaForQuestionAnswering       | 128 |  0.8721  |
|           LayoutLMForMaskedLM           | 16  |  0.8682  |
|            MBartForCausalLM             | 32  |  0.8672  |
|           PegasusForCausalLM            | 32  |  0.8666  |
|    MegatronBertForQuestionAnswering     | 16  |  0.8616  |
|             BertForMaskedLM             | 64  |  0.8443  |
|     DistilBertForQuestionAnswering      | 64  |  0.8442  |
|        BertForQuestionAnswering         | 128 |  0.8413  |
|          DistilBertForMaskedLM          | 64  |  0.8356  |
|             BartForCausalLM             |  4  |  0.8227  |
|      BartForConditionalGeneration       |  2  |  0.8191  |
|       ElectraForQuestionAnswering       | 64  |  0.8126  |
| BlenderbotSmallForConditionalGeneration | 64  |  0.8103  |
|       BlenderbotSmallForCausalLM        | 64  |  0.7972  |
+-----------------------------------------+-----+----------+

Accuracy

+-----------------------------------------+----+----------+
|                  name                   | bs | inductor |
+-----------------------------------------+----+----------+
|            AlbertForMaskedLM            | 1  |   pass   |
|       AlbertForQuestionAnswering        | 1  |   pass   |
|                CamemBert                | 1  |   pass   |
|          AllenaiLongformerBase          | 1  |   pass   |
|             BartForCausalLM             | 1  |   pass   |
|      BartForConditionalGeneration       | 1  |   pass   |
|             BertForMaskedLM             | 1  |   pass   |
|        BertForQuestionAnswering         | 1  |   pass   |
|                 BigBird                 | 1  |   pass   |
|       BlenderbotSmallForCausalLM        | 1  |   pass   |
| BlenderbotSmallForConditionalGeneration | 1  |   pass   |
|           DebertaForMaskedLM            | 1  |   pass   |
|           LayoutLMForMaskedLM           | 1  |   pass   |
|       DebertaForQuestionAnswering       | 1  |   pass   |
|          DistilBertForMaskedLM          | 1  |   pass   |
|     DistilBertForQuestionAnswering      | 1  |   pass   |
|               DistillGPT2               | 1  |   pass   |
|           ElectraForCausalLM            | 1  |   pass   |
|       ElectraForQuestionAnswering       | 1  |   pass   |
|      GPT2ForSequenceClassification      | 1  |   pass   |
|               GoogleFnet                | 1  |   pass   |
|    LayoutLMForSequenceClassification    | 1  |   pass   |
|     M2M100ForConditionalGeneration      | 1  |   pass   |
|            MBartForCausalLM             | 1  |   pass   |
|     PLBartForConditionalGeneration      | 1  |   pass   |
|      MBartForConditionalGeneration      | 1  |   pass   |
|       MT5ForConditionalGeneration       | 1  |   pass   |
|         MegatronBertForCausalLM         | 1  |   pass   |
|    MegatronBertForQuestionAnswering     | 1  |   pass   |
|          MobileBertForMaskedLM          | 1  |   pass   |
|     MobileBertForQuestionAnswering      | 1  |   pass   |
|             OPTForCausalLM              | 1  |   pass   |
|            PLBartForCausalLM            | 1  |   pass   |
|           PegasusForCausalLM            | 1  |   pass   |
|            XLNetLMHeadModel             | 1  |   pass   |
|     PegasusForConditionalGeneration     | 1  |   pass   |
|           RobertaForCausalLM            | 1  |   pass   |
|       RobertaForQuestionAnswering       | 1  |   pass   |
|         Speech2Text2ForCausalLM         | 1  |   pass   |
|       T5ForConditionalGeneration        | 1  |   pass   |
|                 T5Small                 | 1  |   pass   |
|            TrOCRForCausalLM             | 1  |   pass   |
|             XGLMForCausalLM             | 1  |   pass   |
|            YituTechConvBert             | 1  |   pass   |
+-----------------------------------------+----+----------+

Compilation latency (sec)

+-----------------------------------------+-----+----------+
|                  name                   | bs  | inductor |
+-----------------------------------------+-----+----------+
|          MobileBertForMaskedLM          | 32  | 21.5827  |
|            YituTechConvBert             |  1  | 20.8297  |
|     MobileBertForQuestionAnswering      | 64  | 20.8148  |
|          AllenaiLongformerBase          |  1  | 20.7919  |
|           PegasusForCausalLM            | 32  | 20.7313  |
|     PegasusForConditionalGeneration     | 16  | 20.7223  |
|       DebertaForQuestionAnswering       |  8  | 20.5132  |
|           DebertaForMaskedLM            |  4  | 19.8944  |
|                 BigBird                 |  1  | 19.2729  |
|     M2M100ForConditionalGeneration      |  8  | 19.2597  |
|       AlbertForQuestionAnswering        |  4  | 18.5805  |
|      BartForConditionalGeneration       |  2  | 18.2648  |
|      MBartForConditionalGeneration      | 16  |  17.935  |
|             XGLMForCausalLM             |  8  | 17.3405  |
|     DistilBertForQuestionAnswering      | 64  | 16.6965  |
|         MegatronBertForCausalLM         | 16  | 16.4154  |
|    MegatronBertForQuestionAnswering     | 16  | 16.1948  |
|     PLBartForConditionalGeneration      | 16  | 14.9079  |
| BlenderbotSmallForConditionalGeneration | 64  | 14.6041  |
|       MT5ForConditionalGeneration       |  8  | 12.8268  |
|           RobertaForCausalLM            | 64  | 12.7428  |
|       T5ForConditionalGeneration        |  4  | 12.6852  |
|       RobertaForQuestionAnswering       | 128 | 12.6526  |
|           LayoutLMForMaskedLM           | 16  | 12.4495  |
|       ElectraForQuestionAnswering       | 64  | 12.3243  |
|           ElectraForCausalLM            | 32  | 12.2365  |
|             BertForMaskedLM             | 64  | 12.1568  |
|    LayoutLMForSequenceClassification    | 16  | 12.0709  |
|             BartForCausalLM             |  4  | 11.8716  |
|            MBartForCausalLM             | 32  | 11.7556  |
|        BertForQuestionAnswering         | 128 | 11.5338  |
|            TrOCRForCausalLM             | 32  | 11.3487  |
|                 T5Small                 |  1  | 11.3371  |
|      GPT2ForSequenceClassification      |  4  | 11.2746  |
|            AlbertForMaskedLM            |  4  | 11.0849  |
|             OPTForCausalLM              | 32  | 11.0221  |
|               GoogleFnet                |  1  | 10.7448  |
|                CamemBert                |  1  |  10.619  |
|         Speech2Text2ForCausalLM         | 128 | 10.3587  |
|       BlenderbotSmallForCausalLM        | 64  | 10.1789  |
|          DistilBertForMaskedLM          | 64  |  9.5233  |
|            PLBartForCausalLM            | 32  |  9.1554  |
|               DistillGPT2               |  1  |  8.2035  |
|            XLNetLMHeadModel             | 32  | -8.5389  |
+-----------------------------------------+-----+----------+

Peak Memory Compression Ratio

+-----------------------------------------+-----+----------+
|                  name                   | bs  | inductor |
+-----------------------------------------+-----+----------+
|            AlbertForMaskedLM            |  4  |  0.9977  |
|       AlbertForQuestionAnswering        |  4  |  0.9977  |
|             BartForCausalLM             |  4  |  0.9962  |
|       ElectraForQuestionAnswering       | 64  |  0.9962  |
|           RobertaForCausalLM            | 64  |  0.9956  |
|           ElectraForCausalLM            | 32  |  0.9954  |
|       BlenderbotSmallForCausalLM        | 64  |  0.9953  |
|             BertForMaskedLM             | 64  |  0.9952  |
|          DistilBertForMaskedLM          | 64  |  0.9946  |
|            TrOCRForCausalLM             | 32  |  0.9941  |
|      GPT2ForSequenceClassification      |  4  |  0.9941  |
| BlenderbotSmallForConditionalGeneration | 64  |  0.9939  |
|         Speech2Text2ForCausalLM         | 128 |  0.9938  |
|       DebertaForQuestionAnswering       |  8  |  0.9938  |
|           PegasusForCausalLM            | 32  |  0.9935  |
|            MBartForCausalLM             | 32  |  0.9933  |
|       T5ForConditionalGeneration        |  4  |  0.9932  |
|             OPTForCausalLM              | 32  |  0.993   |
|           LayoutLMForMaskedLM           | 16  |  0.993   |
|    LayoutLMForSequenceClassification    | 16  |  0.9927  |
|            PLBartForCausalLM            | 32  |  0.9925  |
|     PegasusForConditionalGeneration     | 16  |  0.9921  |
|      BartForConditionalGeneration       |  2  |  0.9919  |
|                 BigBird                 |  1  |  0.9915  |
|           DebertaForMaskedLM            |  4  |  0.9913  |
|       RobertaForQuestionAnswering       | 128 |  0.9912  |
|      MBartForConditionalGeneration      | 16  |  0.9876  |
|         MegatronBertForCausalLM         | 16  |  0.9867  |
|               GoogleFnet                |  1  |  0.9838  |
|     PLBartForConditionalGeneration      | 16  |  0.9834  |
|               DistillGPT2               |  1  |  0.9828  |
|        BertForQuestionAnswering         | 128 |  0.9817  |
|          MobileBertForMaskedLM          | 32  |  0.9802  |
|     DistilBertForQuestionAnswering      | 64  |  0.9799  |
|            XLNetLMHeadModel             | 32  |  0.9733  |
|             XGLMForCausalLM             |  8  |  0.964   |
|    MegatronBertForQuestionAnswering     | 16  |  0.9603  |
|            YituTechConvBert             |  1  |  0.9454  |
|                 T5Small                 |  1  |  0.9432  |
|     M2M100ForConditionalGeneration      |  8  |  0.9419  |
|          AllenaiLongformerBase          |  1  |  0.9138  |
|     MobileBertForQuestionAnswering      | 64  |  0.8877  |
|                CamemBert                |  1  |  0.8388  |
|       MT5ForConditionalGeneration       |  8  |  0.7901  |
+-----------------------------------------+-----+----------+

timm_models suite with float32 precision

see more

Performance speedup

+---------------------------------+-----+----------+
|              name               | bs  | inductor |
+---------------------------------+-----+----------+
|          pnasnet5large          | 16  |  1.4327  |
|          inception_v3           | 128 |  1.2815  |
|       gluon_inception_v3        | 128 |  1.2746  |
|        adv_inception_v3         | 128 |   1.26   |
|           fbnetc_100            | 128 |  1.1964  |
|           res2next50            | 128 |  1.1947  |
|           mnasnet_100           | 128 |  1.1861  |
|          spnasnet_100           | 128 |  1.1838  |
|        res2net50_14w_8s         | 128 |  1.1719  |
|        ese_vovnet19b_dw         | 128 |  1.1705  |
|      mobilenetv3_large_100      | 128 |  1.1667  |
|             dpn107              | 32  |  1.1605  |
|             dla102              | 128 |  1.1539  |
|          ghostnet_100           | 128 |  1.1521  |
|            lcnet_050            | 128 |  1.1492  |
|            hrnet_w18            | 128 |  1.1482  |
|        res2net101_26w_4s        | 64  |  1.1464  |
|           volo_d1_224           | 64  |  1.1299  |
|            gernet_l             | 128 |  1.122   |
|            repvgg_a2            | 128 |  1.1182  |
|        gluon_xception65         | 32  |  1.1142  |
|         visformer_small         | 128 |  1.093   |
|           selecsls42b           | 128 |  1.0929  |
|            fbnetv3_b            | 128 |  1.0894  |
|     swsl_resnext101_32x16d      | 32  |  1.0821  |
|         mobilenetv2_100         | 128 |  1.0725  |
|           regnety_002           | 128 |  1.0644  |
|           tf_mixnet_l           | 128 |  1.0574  |
|          cspdarknet53           | 64  |  1.0558  |
|           resnest101e           | 64  |  1.0425  |
|      xcit_large_24_p8_224       |  5  |  1.0173  |
|          gmixer_24_224          | 128 |  1.0126  |
|            mixnet_l             | 128 |  1.0071  |
|          cait_m36_384           |  4  |  0.9831  |
|            nfnet_l0             | 128 |  0.9106  |
|  swin_base_patch4_window7_224   | 64  |  0.8936  |
|      beit_base_patch16_224      | 64  |  0.8895  |
|           rexnet_100            | 128 |  0.8856  |
|          convnext_base          | 64  |  0.8827  |
|           dm_nfnet_f0           | 128 |  0.879   |
|           convit_base           | 64  |  0.8711  |
|         poolformer_m36          | 64  |  0.8597  |
|       tf_efficientnet_b0        | 128 |  0.8576  |
|      vit_base_patch16_224       | 64  |  0.8576  |
| deit_base_distilled_patch16_224 | 64  |  0.8571  |
|           mobilevit_s           | 64  |  0.8447  |
|          jx_nest_base           | 32  |  0.8343  |
|          resmlp_12_224          | 128 |  0.8309  |
|            tinynet_a            | 128 |  0.8178  |
|        tnt_s_patch16_224        | 128 |  0.8126  |
|            pit_b_224            | 64  |  0.8018  |
|          mixer_b16_224          | 128 |  0.7926  |
|         coat_lite_mini          | 128 |  0.7568  |
|        twins_pcpvt_base         | 64  |  0.7359  |
|         crossvit_9_240          | 128 |  0.7294  |
|          gmlp_s16_224           | 128 |  0.5807  |
|        sebotnet33ts_256         |  0  |   0.0    |
|       eca_botnext26ts_256       |  0  |   0.0    |
|        eca_halonext26ts         |  0  |   0.0    |
|          botnet26t_256          |  0  |   0.0    |
|        convmixer_768_32         |  0  |   0.0    |
+---------------------------------+-----+----------+

Accuracy

+---------------------------------+----+---------------+
|              name               | bs |   inductor    |
+---------------------------------+----+---------------+
|        adv_inception_v3         | 2  |     pass      |
|             dpn107              | 2  |     pass      |
|      beit_base_patch16_224      | 2  |     pass      |
|          mixer_b16_224          | 2  |     pass      |
|        ese_vovnet19b_dw         | 2  |     pass      |
|         coat_lite_mini          | 2  |     pass      |
|           convit_base           | 2  |     pass      |
|          convnext_base          | 2  |     pass      |
|         crossvit_9_240          | 2  |     pass      |
|          cspdarknet53           | 2  |     pass      |
| deit_base_distilled_patch16_224 | 2  |     pass      |
|             dla102              | 2  |     pass      |
|           dm_nfnet_f0           | 2  |     pass      |
|            lcnet_050            | 2  |     pass      |
|           volo_d1_224           | 2  |     pass      |
|      xcit_large_24_p8_224       | 2  |     pass      |
|           fbnetc_100            | 2  |     pass      |
|        gluon_xception65         | 2  |     pass      |
|          jx_nest_base           | 2  |     pass      |
|          inception_v3           | 2  |     pass      |
|            hrnet_w18            | 2  |     pass      |
|          gmlp_s16_224           | 2  |     pass      |
|          gmixer_24_224          | 2  |     pass      |
|       gluon_inception_v3        | 2  |     pass      |
|            gernet_l             | 2  |     pass      |
|            fbnetv3_b            | 2  |     pass      |
|           mnasnet_100           | 2  |     pass      |
|            mixnet_l             | 2  |     pass      |
|      vit_base_patch16_224       | 2  |     pass      |
|           res2next50            | 2  |     pass      |
|         mobilenetv2_100         | 2  |     pass      |
|      mobilenetv3_large_100      | 2  |     pass      |
|           mobilevit_s           | 2  |     pass      |
|            nfnet_l0             | 2  |     pass      |
|            pit_b_224            | 2  |     pass      |
|          pnasnet5large          | 2  |     pass      |
|         poolformer_m36          | 2  |     pass      |
|           regnety_002           | 2  |     pass      |
|            repvgg_a2            | 2  |     pass      |
|         visformer_small         | 2  |     pass      |
|        res2net50_14w_8s         | 2  |     pass      |
|        res2net101_26w_4s        | 2  |     pass      |
|          resmlp_12_224          | 2  |     pass      |
|       tf_efficientnet_b0        | 2  |     pass      |
|        twins_pcpvt_base         | 2  |     pass      |
|        tnt_s_patch16_224        | 2  |     pass      |
|           resnest101e           | 2  |     pass      |
|           tf_mixnet_l           | 2  |     pass      |
|            tinynet_a            | 2  |     pass      |
|     swsl_resnext101_32x16d      | 2  |     pass      |
|  swin_base_patch4_window7_224   | 2  |     pass      |
|          spnasnet_100           | 2  |     pass      |
|           selecsls42b           | 2  |     pass      |
|           rexnet_100            | 2  |     pass      |
|        eca_halonext26ts         | 2  |  fail_to_run  |
|        convmixer_768_32         | 2  |  fail_to_run  |
|        sebotnet33ts_256         | 2  |  fail_to_run  |
|          botnet26t_256          | 2  |  fail_to_run  |
|       eca_botnext26ts_256       | 2  |  fail_to_run  |
|          ghostnet_100           | 2  | fail_accuracy |
|          cait_m36_384           | 2  | fail_accuracy |
+---------------------------------+----+---------------+

Compilation latency (sec)

+---------------------------------+-----+----------+
|              name               | bs  | inductor |
+---------------------------------+-----+----------+
|            hrnet_w18            | 128 | 38.6761  |
|          pnasnet5large          | 16  | 30.2884  |
|  swin_base_patch4_window7_224   | 64  | 24.3665  |
|     swsl_resnext101_32x16d      | 32  | 22.5364  |
|          cait_m36_384           |  4  | 22.3963  |
|        twins_pcpvt_base         | 64  | 21.5813  |
|           tf_mixnet_l           | 128 | 21.4847  |
|           dm_nfnet_f0           | 128 | 21.4823  |
|           resnest101e           | 64  | 21.2826  |
|        res2net50_14w_8s         | 128 | 21.0787  |
|             dla102              | 128 | 20.9507  |
|         poolformer_m36          | 64  | 20.5791  |
|           mobilevit_s           | 64  | 20.5489  |
|      xcit_large_24_p8_224       |  5  | 20.2446  |
|        res2net101_26w_4s        | 64  | 20.0055  |
|            mixnet_l             | 128 | 19.8109  |
|        tnt_s_patch16_224        | 128 | 19.3401  |
|            fbnetv3_b            | 128 | 18.1484  |
|          gmlp_s16_224           | 128 |  18.061  |
|           regnety_002           | 128 | 18.0097  |
|             dpn107              | 32  | 17.9336  |
|           rexnet_100            | 128 | 17.8279  |
|            nfnet_l0             | 128 | 17.8076  |
|          jx_nest_base           | 32  | 17.2234  |
|        gluon_xception65         | 32  |  16.187  |
|           res2next50            | 128 | 15.6389  |
|          convnext_base          | 64  | 15.6367  |
|         crossvit_9_240          | 128 | 15.5356  |
|           volo_d1_224           | 64  | 15.4737  |
|       tf_efficientnet_b0        | 128 | 15.2815  |
|            tinynet_a            | 128 | 15.0834  |
|        adv_inception_v3         | 128 | 14.7912  |
|          inception_v3           | 128 | 14.7196  |
|       gluon_inception_v3        | 128 | 14.6679  |
|          ghostnet_100           | 128 | 14.5108  |
|         coat_lite_mini          | 128 | 14.4649  |
|           convit_base           | 64  | 14.0784  |
|            pit_b_224            | 64  |  14.03   |
|          cspdarknet53           | 64  | 13.4574  |
|          gmixer_24_224          | 128 | 12.8121  |
|          mixer_b16_224          | 128 | 12.6285  |
|      mobilenetv3_large_100      | 128 | 12.2343  |
|      beit_base_patch16_224      | 64  | 12.1593  |
|         mobilenetv2_100         | 128 | 12.1048  |
|           fbnetc_100            | 128 | 11.9571  |
|          spnasnet_100           | 128 | 11.8834  |
| deit_base_distilled_patch16_224 | 64  |  11.866  |
|         visformer_small         | 128 | 11.8494  |
|      vit_base_patch16_224       | 64  | 11.7031  |
|            gernet_l             | 128 | 11.1115  |
|           mnasnet_100           | 128 | 10.9839  |
|            repvgg_a2            | 128 |  10.406  |
|        ese_vovnet19b_dw         | 128 |  9.8925  |
|           selecsls42b           | 128 |  9.7574  |
|          resmlp_12_224          | 128 |  9.6532  |
|            lcnet_050            | 128 |  9.0859  |
|          botnet26t_256          |  0  |   nan    |
|        convmixer_768_32         |  0  |   nan    |
|       eca_botnext26ts_256       |  0  |   nan    |
|        eca_halonext26ts         |  0  |   nan    |
|        sebotnet33ts_256         |  0  |   nan    |
+---------------------------------+-----+----------+

Peak Memory Compression Ratio

+---------------------------------+-----+----------+
|              name               | bs  | inductor |
+---------------------------------+-----+----------+
|          mixer_b16_224          | 128 |  0.9979  |
|        ese_vovnet19b_dw         | 128 |  0.9974  |
|           res2next50            | 128 |  0.9972  |
|         visformer_small         | 128 |  0.9971  |
|     swsl_resnext101_32x16d      | 32  |  0.997   |
|         coat_lite_mini          | 128 |  0.9969  |
|           dm_nfnet_f0           | 128 |  0.9968  |
|           tf_mixnet_l           | 128 |  0.9966  |
|            mixnet_l             | 128 |  0.9965  |
| deit_base_distilled_patch16_224 | 64  |  0.9965  |
|           convit_base           | 64  |  0.9963  |
|      vit_base_patch16_224       | 64  |  0.9963  |
|            gernet_l             | 128 |  0.9962  |
|        adv_inception_v3         | 128 |  0.9962  |
|      beit_base_patch16_224      | 64  |  0.9961  |
|          resmlp_12_224          | 128 |  0.996   |
|        gluon_xception65         | 32  |  0.996   |
|          gmixer_24_224          | 128 |  0.996   |
|            fbnetv3_b            | 128 |  0.9959  |
|       tf_efficientnet_b0        | 128 |  0.9959  |
|          convnext_base          | 64  |  0.9958  |
|          cspdarknet53           | 64  |  0.9958  |
|           resnest101e           | 64  |  0.9958  |
|        res2net50_14w_8s         | 128 |  0.9957  |
|            pit_b_224            | 64  |  0.9957  |
|           selecsls42b           | 128 |  0.9957  |
|           mobilevit_s           | 64  |  0.9956  |
|          gmlp_s16_224           | 128 |  0.9955  |
|            nfnet_l0             | 128 |  0.9948  |
|           fbnetc_100            | 128 |  0.9948  |
|          spnasnet_100           | 128 |  0.9945  |
|           rexnet_100            | 128 |  0.9945  |
|           mnasnet_100           | 128 |  0.9944  |
|        tnt_s_patch16_224        | 128 |  0.9942  |
|             dpn107              | 32  |  0.994   |
|            repvgg_a2            | 128 |  0.9939  |
|      mobilenetv3_large_100      | 128 |  0.9938  |
|         mobilenetv2_100         | 128 |  0.9936  |
|       gluon_inception_v3        | 128 |  0.9935  |
|  swin_base_patch4_window7_224   | 64  |  0.9935  |
|          inception_v3           | 128 |  0.9934  |
|         poolformer_m36          | 64  |  0.9933  |
|            tinynet_a            | 128 |  0.9928  |
|          cait_m36_384           |  4  |  0.9927  |
|          jx_nest_base           | 32  |  0.9926  |
|          ghostnet_100           | 128 |  0.9921  |
|        twins_pcpvt_base         | 64  |  0.992   |
|        res2net101_26w_4s        | 64  |  0.992   |
|          pnasnet5large          | 16  |  0.9919  |
|            hrnet_w18            | 128 |  0.9917  |
|         crossvit_9_240          | 128 |  0.9915  |
|           volo_d1_224           | 64  |  0.9911  |
|      xcit_large_24_p8_224       |  5  |  0.9894  |
|           regnety_002           | 128 |  0.9887  |
|            lcnet_050            | 128 |  0.9881  |
|             dla102              | 128 |  0.9829  |
|          botnet26t_256          |  0  |   nan    |
|        convmixer_768_32         |  0  |   nan    |
|       eca_botnext26ts_256       |  0  |   nan    |
|        eca_halonext26ts         |  0  |   nan    |
|        sebotnet33ts_256         |  0  |   nan    |
+---------------------------------+-----+----------+

[yudongsi]

Performance Dashboard for float32 precision -- Single-core Single-thread

Executive Summary

see more

We evaluate different backends across three benchmark suites - torchbench, huggingface and timm. We run these experiments on Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz. Each experiment runs one iteration of forward pass. For accuracy, we check the numerical correctness of forward pass outputs by comparing with native pytorch. We measure speedup by normalizing against the performance of native pytorch. We report mean compilation latency numbers and peak memory footprint reduction ratio.

Caveats

1. Batch size has been reduced to workaround OOM errors. Work is in progress to reduce peak memory footprint.
2. Experiments do not cover dynamic shapes.
3. Experimental setup does not have optimizer.

To measure performance, compilation latency and memory footprint reduction, we remove the models that fail accuracy checks.

Passrate

+----------+------------+-------------+-------------+
| Compiler | torchbench | huggingface | timm_models |
+----------+------------+-------------+-------------+
| inductor | 91%, 50/55 | 100%, 44/44 | 89%, 54/61  |
+----------+------------+-------------+-------------+

Geometric mean speedup

+----------+------------+-------------+-------------+
| Compiler | torchbench | huggingface | timm_models |
+----------+------------+-------------+-------------+
| inductor |   1.07x    |     1.00x   |    1.10x    |
+----------+------------+-------------+-------------+

Mean compilation time (seconds)

+----------+------------+-------------+-------------+
| Compiler | torchbench | huggingface | timm_models |
+----------+------------+-------------+-------------+
| inductor |    18.23   |    18.39    |    21.87    |
+----------+------------+-------------+-------------+

Peak memory footprint compression ratio (higher is better)

+----------+------------+-------------+-------------+
| Compiler | torchbench | huggingface | timm_models |
+----------+------------+-------------+-------------+
| inductor |   1.18x    |    1.32x    |    1.11x    |
+----------+------------+-------------+-------------+

torchbench suite with float32 precision

see more

Performance speedup

+-----------------------------------+----+----------+
|               name                | bs | inductor |
+-----------------------------------+----+----------+
|        shufflenet_v2_x1_0         | 1  |  1.6303  |
|           timm_resnest            | 1  |  1.3047  |
| attention_is_all_you_need_pytorch | 1  |  1.2927  |
|           mobilenet_v2            | 1  |  1.2252  |
|           squeezenet1_1           | 1  |  1.2142  |
|   pytorch_CycleGAN_and_pix2pix    | 1  |  1.1865  |
|            densenet121            | 1  |  1.1803  |
|            mnasnet1_0             | 1  |  1.1663  |
|        mobilenet_v3_large         | 1  |  1.1543  |
|       functorch_dp_cifar10        | 1  |  1.1534  |
|          resnext50_32x4d          | 1  |  1.1496  |
|             resnet18              | 1  |  1.1423  |
|             resnet50              | 1  |  1.136   |
|          vision_maskrcnn          | 1  |  1.1325  |
|            timm_vovnet            | 1  |  1.1164  |
|     detectron2_fcos_r_50_fpn      | 1  |  1.0986  |
|               vgg16               | 1  |  1.0902  |
|              alexnet              | 1  |  1.0841  |
|                drq                | 1  |  1.0802  |
|           pytorch_unet            | 1  |  1.0801  |
|               dlrm                | 1  |   1.07   |
|            timm_regnet            | 1  |  1.0692  |
|          LearningToPaint          | 1  |  1.0669  |
|              yolov3               | 1  |  1.0582  |
|            Super_SloMo            | 1  |  1.0568  |
|               dcgan               | 1  |   1.03   |
|          pytorch_stargan          | 16 |  1.0153  |
|        Background_Matting         | 1  |  1.015   |
|              demucs               | 1  |  1.0007  |
|      resnet50_quantized_qat       | 1  |  0.9971  |
|            tts_angular            | 1  |  0.9956  |
|            timm_nfnet             | 1  |  0.994   |
|            hf_BigBird             | 1  |  0.9933  |
|    mobilenet_v2_quantized_qat     | 1  |  0.9929  |
|      nvidia_deeprecommender       | 1  |  0.9801  |
|              hf_GPT2              | 1  |  0.8869  |
|             hf_Albert             | 1  |  0.8366  |
|           hf_Longformer           | 1  |  0.8335  |
|            hf_Reformer            | 1  |  0.824   |
|           lennard_jones           | 1  |  0.8225  |
|           BERT_pytorch            | 1  |  0.8156  |
|            hf_T5_large            | 1  |  0.8146  |
|   timm_vision_transformer_large   | 1  |  0.8083  |
|           hf_GPT2_large           | 1  |  0.7638  |
|               hf_T5               | 1  |  0.7614  |
|           hf_DistilBert           | 1  |  0.7511  |
|              hf_Bert              | 1  |  0.719   |
|              hf_Bart              | 1  |  0.7189  |
|            hf_T5_base             | 1  |  0.6985  |
|      timm_vision_transformer      | 1  |  0.6076  |
|         timm_efficientnet         | 1  |  0.5181  |
|           fastNLP_Bert            | 0  |   0.0    |
|             tacotron2             | 0  |   0.0    |
|        speech_transformer         | 0  |   0.0    |
|         soft_actor_critic         | 0  |   0.0    |
+-----------------------------------+----+----------+

Accuracy

+-----------------------------------+----+------------------+
|               name                | bs |     inductor     |
+-----------------------------------+----+------------------+
|            hf_T5_large            | 1  | pass_due_to_skip |
|           hf_GPT2_large           | 1  | pass_due_to_skip |
|   timm_vision_transformer_large   | 1  | pass_due_to_skip |
|               hf_T5               | 1  |       pass       |
|               dlrm                | 1  |       pass       |
|            hf_T5_base             | 1  |       pass       |
|          LearningToPaint          | 1  |       pass       |
|            Super_SloMo            | 1  |       pass       |
|              alexnet              | 1  |       pass       |
| attention_is_all_you_need_pytorch | 1  |       pass       |
|               dcgan               | 1  |       pass       |
|              demucs               | 1  |       pass       |
|            densenet121            | 1  |       pass       |
|     detectron2_fcos_r_50_fpn      | 1  |       pass       |
|                drq                | 1  |       pass       |
|            hf_Reformer            | 1  |       pass       |
|            mnasnet1_0             | 1  |       pass       |
|       functorch_dp_cifar10        | 1  |       pass       |
|             hf_Albert             | 1  |       pass       |
|              hf_Bart              | 1  |       pass       |
|              hf_Bert              | 1  |       pass       |
|            hf_BigBird             | 1  |       pass       |
|           hf_DistilBert           | 1  |       pass       |
|              hf_GPT2              | 1  |       pass       |
|           hf_Longformer           | 1  |       pass       |
|              yolov3               | 1  |       pass       |
|           lennard_jones           | 1  |       pass       |
|          resnext50_32x4d          | 1  |       pass       |
|           BERT_pytorch            | 1  |       pass       |
|      resnet50_quantized_qat       | 1  |       pass       |
|           mobilenet_v2            | 1  |       pass       |
|    mobilenet_v2_quantized_qat     | 1  |       pass       |
|        mobilenet_v3_large         | 1  |       pass       |
|      nvidia_deeprecommender       | 1  |       pass       |
|   pytorch_CycleGAN_and_pix2pix    | 1  |       pass       |
|          pytorch_stargan          | 16 |       pass       |
|           pytorch_unet            | 1  |       pass       |
|               vgg16               | 1  |       pass       |
|             resnet50              | 1  |       pass       |
|             resnet18              | 1  |       pass       |
|        Background_Matting         | 1  |       pass       |
|        shufflenet_v2_x1_0         | 1  |       pass       |
|           squeezenet1_1           | 1  |       pass       |
|         timm_efficientnet         | 1  |       pass       |
|            timm_nfnet             | 1  |       pass       |
|            timm_regnet            | 1  |       pass       |
|           timm_resnest            | 1  |       pass       |
|      timm_vision_transformer      | 1  |       pass       |
|            timm_vovnet            | 1  |       pass       |
|            tts_angular            | 1  |       pass       |
|         soft_actor_critic         | 0  |      0.0000      |
|           fastNLP_Bert            | 0  |      0.0000      |
|             tacotron2             | 0  |      0.0000      |
|          vision_maskrcnn          | 0  |      0.0000      |
|        speech_transformer         | 0  |      0.0000      |
+-----------------------------------+----+------------------+

Compilation latency (sec)

+-----------------------------------+----+----------+
|               name                | bs | inductor |
+-----------------------------------+----+----------+
|          vision_maskrcnn          | 1  | 67.9055  |
|     detectron2_fcos_r_50_fpn      | 1  | 65.0179  |
|            hf_T5_base             | 1  | 61.0279  |
|            hf_T5_large            | 1  | 55.3729  |
|           hf_GPT2_large           | 1  | 44.9441  |
|            densenet121            | 1  | 32.9563  |
|           hf_Longformer           | 1  | 31.9838  |
|            timm_nfnet             | 1  | 26.7587  |
|            hf_BigBird             | 1  | 26.0525  |
|              yolov3               | 1  | 25.8128  |
|   timm_vision_transformer_large   | 1  | 25.5415  |
|         timm_efficientnet         | 1  | 24.3729  |
|            Super_SloMo            | 1  | 23.1083  |
|        mobilenet_v3_large         | 1  | 22.8879  |
|            timm_vovnet            | 1  | 22.6841  |
|            timm_regnet            | 1  | 19.7768  |
|              hf_Bart              | 1  | 17.1645  |
|            mnasnet1_0             | 1  |  17.093  |
|        shufflenet_v2_x1_0         | 1  | 17.0703  |
|               hf_T5               | 1  | 16.8451  |
|           mobilenet_v2            | 1  | 16.7573  |
|            hf_Reformer            | 1  | 16.7426  |
|           timm_resnest            | 1  |  15.893  |
|        Background_Matting         | 1  | 15.1108  |
|             resnet50              | 1  | 15.0244  |
|          resnext50_32x4d          | 1  | 15.0111  |
|              hf_GPT2              | 1  | 14.6155  |
|           BERT_pytorch            | 1  | 13.7351  |
|              hf_Bert              | 1  | 13.1208  |
|      timm_vision_transformer      | 1  | 13.1204  |
| attention_is_all_you_need_pytorch | 1  | 12.8987  |
|       functorch_dp_cifar10        | 1  | 12.6869  |
|             hf_Albert             | 1  | 12.3245  |
|           pytorch_unet            | 1  |  12.05   |
|             resnet18              | 1  | 11.8429  |
|           hf_DistilBert           | 1  |  11.227  |
|          LearningToPaint          | 1  | 11.1157  |
|           squeezenet1_1           | 1  | 11.0953  |
|   pytorch_CycleGAN_and_pix2pix    | 1  | 10.7537  |
|          pytorch_stargan          | 16 |  10.425  |
|               vgg16               | 1  |  9.3719  |
|               dlrm                | 1  |  9.1913  |
|      nvidia_deeprecommender       | 1  |  9.0608  |
|                drq                | 1  |  8.9684  |
|            tts_angular            | 1  |  8.1898  |
|              alexnet              | 1  |  7.8231  |
|           lennard_jones           | 1  |  7.7809  |
|               dcgan               | 1  |  7.5431  |
|              demucs               | 1  |  0.9926  |
|    mobilenet_v2_quantized_qat     | 1  |  0.1719  |
|      resnet50_quantized_qat       | 1  |  0.1575  |
|           fastNLP_Bert            | 0  |   nan    |
|         soft_actor_critic         | 0  |   nan    |
|        speech_transformer         | 0  |   nan    |
|             tacotron2             | 0  |   nan    |
+-----------------------------------+----+----------+

Peak Memory Compression Ratio

+-----------------------------------+----+----------+
|               name                | bs | inductor |
+-----------------------------------+----+----------+
|            hf_T5_base             | 1  |  4.0697  |
|           pytorch_unet            | 1  |  2.4777  |
|            hf_BigBird             | 1  |  2.1386  |
|           hf_GPT2_large           | 1  |  1.9364  |
|            Super_SloMo            | 1  |  1.6529  |
|            hf_T5_large            | 1  |  1.5674  |
|           hf_Longformer           | 1  |  1.4812  |
|          vision_maskrcnn          | 1  |  1.4019  |
|              yolov3               | 1  |  1.2458  |
|          LearningToPaint          | 1  |  1.2383  |
|           hf_DistilBert           | 1  |  1.2054  |
|            timm_nfnet             | 1  |  1.1525  |
|             hf_Albert             | 1  |  1.1471  |
|            timm_regnet            | 1  |  1.144   |
|   timm_vision_transformer_large   | 1  |  1.1144  |
|          resnext50_32x4d          | 1  |  1.1056  |
|             resnet50              | 1  |  1.0811  |
|                drq                | 1  |  1.0667  |
|           mobilenet_v2            | 1  |  1.0624  |
|            timm_vovnet            | 1  |  1.0523  |
|         timm_efficientnet         | 1  |  1.0511  |
|           timm_resnest            | 1  |  1.0487  |
|      timm_vision_transformer      | 1  |  1.0453  |
|              hf_Bart              | 1  |  1.0411  |
|            mnasnet1_0             | 1  |  1.0159  |
|              demucs               | 1  |  0.9985  |
|               dlrm                | 1  |  0.9976  |
|             resnet18              | 1  |  0.9969  |
|      nvidia_deeprecommender       | 1  |  0.9965  |
|            tts_angular            | 1  |  0.9958  |
|      resnet50_quantized_qat       | 1  |  0.9956  |
|        Background_Matting         | 1  |  0.9937  |
|    mobilenet_v2_quantized_qat     | 1  |  0.9934  |
|          pytorch_stargan          | 16 |  0.9883  |
| attention_is_all_you_need_pytorch | 1  |  0.9837  |
|        mobilenet_v3_large         | 1  |  0.9822  |
|   pytorch_CycleGAN_and_pix2pix    | 1  |  0.9813  |
|              alexnet              | 1  |  0.9807  |
|           lennard_jones           | 1  |  0.9804  |
|     detectron2_fcos_r_50_fpn      | 1  |  0.9786  |
|               vgg16               | 1  |  0.9737  |
|       functorch_dp_cifar10        | 1  |  0.9703  |
|           squeezenet1_1           | 1  |  0.9689  |
|              hf_GPT2              | 1  |  0.9674  |
|        shufflenet_v2_x1_0         | 1  |  0.9536  |
|               dcgan               | 1  |  0.9478  |
|              hf_Bert              | 1  |  0.9125  |
|           BERT_pytorch            | 1  |  0.8995  |
|               hf_T5               | 1  |  0.808   |
|            densenet121            | 1  |  0.7885  |
|            hf_Reformer            | 1  |  0.7624  |
|           fastNLP_Bert            | 0  |   nan    |
|         soft_actor_critic         | 0  |   nan    |
|        speech_transformer         | 0  |   nan    |
|             tacotron2             | 0  |   nan    |
+-----------------------------------+----+----------+

huggingface suite with float32 precision

see more

Performance speedup

+-----------------------------------------+----+----------+
|                  name                   | bs | inductor |
+-----------------------------------------+----+----------+
|     MobileBertForQuestionAnswering      | 1  |  1.0915  |
|            XLNetLMHeadModel             | 1  |  1.0461  |
|       AlbertForQuestionAnswering        | 1  |  0.9584  |
|          MobileBertForMaskedLM          | 1  |  0.9445  |
|            AlbertForMaskedLM            | 1  |  0.9374  |
|                 BigBird                 | 1  |  0.9061  |
|     M2M100ForConditionalGeneration      | 1  |  0.9058  |
|             OPTForCausalLM              | 1  |  0.881   |
|               GoogleFnet                | 1  |  0.8584  |
|            YituTechConvBert             | 1  |  0.824   |
|         Speech2Text2ForCausalLM         | 1  |  0.823   |
|       DebertaForQuestionAnswering       | 1  |  0.814   |
|      MBartForConditionalGeneration      | 1  |  0.8049  |
|    MegatronBertForQuestionAnswering     | 1  |  0.8045  |
|     PegasusForConditionalGeneration     | 1  |  0.8037  |
|         MegatronBertForCausalLM         | 1  |  0.7996  |
|          AllenaiLongformerBase          | 1  |  0.7925  |
|       RobertaForQuestionAnswering       | 1  |  0.7884  |
|            MBartForCausalLM             | 1  |  0.7875  |
|            TrOCRForCausalLM             | 1  |  0.7868  |
|           PegasusForCausalLM            | 1  |  0.7839  |
|             XGLMForCausalLM             | 1  |  0.7793  |
|           DebertaForMaskedLM            | 1  |  0.7789  |
|           RobertaForCausalLM            | 1  |  0.7719  |
|     DistilBertForQuestionAnswering      | 1  |  0.7685  |
|     PLBartForConditionalGeneration      | 1  |  0.768   |
|        BertForQuestionAnswering         | 1  |  0.7573  |
|          DistilBertForMaskedLM          | 1  |  0.7509  |
|             BertForMaskedLM             | 1  |  0.7464  |
|               DistillGPT2               | 1  |  0.7368  |
|            PLBartForCausalLM            | 1  |  0.7231  |
|       MT5ForConditionalGeneration       | 1  |  0.7122  |
|    LayoutLMForSequenceClassification    | 1  |  0.6877  |
| BlenderbotSmallForConditionalGeneration | 1  |  0.6839  |
|           LayoutLMForMaskedLM           | 1  |  0.6829  |
|      GPT2ForSequenceClassification      | 1  |  0.6814  |
|                CamemBert                | 1  |  0.675   |
|             BartForCausalLM             | 1  |  0.6741  |
|       BlenderbotSmallForCausalLM        | 1  |  0.6694  |
|       T5ForConditionalGeneration        | 1  |  0.6436  |
|                 T5Small                 | 1  |  0.6422  |
|      BartForConditionalGeneration       | 1  |  0.6387  |
|       ElectraForQuestionAnswering       | 1  |  0.5124  |
|           ElectraForCausalLM            | 1  |  0.494   |
+-----------------------------------------+----+----------+

Accuracy

+-----------------------------------------+----+----------+
|                  name                   | bs | inductor |
+-----------------------------------------+----+----------+
|            AlbertForMaskedLM            | 1  |   pass   |
|       AlbertForQuestionAnswering        | 1  |   pass   |
|                CamemBert                | 1  |   pass   |
|          AllenaiLongformerBase          | 1  |   pass   |
|             BartForCausalLM             | 1  |   pass   |
|      BartForConditionalGeneration       | 1  |   pass   |
|             BertForMaskedLM             | 1  |   pass   |
|        BertForQuestionAnswering         | 1  |   pass   |
|                 BigBird                 | 1  |   pass   |
|       BlenderbotSmallForCausalLM        | 1  |   pass   |
| BlenderbotSmallForConditionalGeneration | 1  |   pass   |
|           DebertaForMaskedLM            | 1  |   pass   |
|           LayoutLMForMaskedLM           | 1  |   pass   |
|       DebertaForQuestionAnswering       | 1  |   pass   |
|          DistilBertForMaskedLM          | 1  |   pass   |
|     DistilBertForQuestionAnswering      | 1  |   pass   |
|               DistillGPT2               | 1  |   pass   |
|           ElectraForCausalLM            | 1  |   pass   |
|       ElectraForQuestionAnswering       | 1  |   pass   |
|      GPT2ForSequenceClassification      | 1  |   pass   |
|               GoogleFnet                | 1  |   pass   |
|    LayoutLMForSequenceClassification    | 1  |   pass   |
|     M2M100ForConditionalGeneration      | 1  |   pass   |
|            MBartForCausalLM             | 1  |   pass   |
|     PLBartForConditionalGeneration      | 1  |   pass   |
|      MBartForConditionalGeneration      | 1  |   pass   |
|       MT5ForConditionalGeneration       | 1  |   pass   |
|         MegatronBertForCausalLM         | 1  |   pass   |
|    MegatronBertForQuestionAnswering     | 1  |   pass   |
|          MobileBertForMaskedLM          | 1  |   pass   |
|     MobileBertForQuestionAnswering      | 1  |   pass   |
|             OPTForCausalLM              | 1  |   pass   |
|            PLBartForCausalLM            | 1  |   pass   |
|           PegasusForCausalLM            | 1  |   pass   |
|            XLNetLMHeadModel             | 1  |   pass   |
|     PegasusForConditionalGeneration     | 1  |   pass   |
|           RobertaForCausalLM            | 1  |   pass   |
|       RobertaForQuestionAnswering       | 1  |   pass   |
|         Speech2Text2ForCausalLM         | 1  |   pass   |
|       T5ForConditionalGeneration        | 1  |   pass   |
|                 T5Small                 | 1  |   pass   |
|            TrOCRForCausalLM             | 1  |   pass   |
|             XGLMForCausalLM             | 1  |   pass   |
|            YituTechConvBert             | 1  |   pass   |
+-----------------------------------------+----+----------+

Compilation latency (sec)

+-----------------------------------------+----+----------+
|                  name                   | bs | inductor |
+-----------------------------------------+----+----------+
|      BartForConditionalGeneration       | 1  | 41.6146  |
|          AllenaiLongformerBase          | 1  | 27.9724  |
|          MobileBertForMaskedLM          | 1  | 24.7466  |
|     MobileBertForQuestionAnswering      | 1  | 24.6238  |
|     PegasusForConditionalGeneration     | 1  | 23.6489  |
|     M2M100ForConditionalGeneration      | 1  | 23.5255  |
|      MBartForConditionalGeneration      | 1  | 22.7092  |
|           DebertaForMaskedLM            | 1  | 22.4907  |
|                 BigBird                 | 1  | 22.4712  |
|       MT5ForConditionalGeneration       | 1  | 22.4523  |
|             XGLMForCausalLM             | 1  | 22.1615  |
|       T5ForConditionalGeneration        | 1  | 22.1289  |
|       DebertaForQuestionAnswering       | 1  | 21.9579  |
|                 T5Small                 | 1  |  21.842  |
|             BartForCausalLM             | 1  | 21.1736  |
|       BlenderbotSmallForCausalLM        | 1  | 20.9169  |
|         MegatronBertForCausalLM         | 1  | 20.7428  |
|    LayoutLMForSequenceClassification    | 1  | 20.6166  |
|            XLNetLMHeadModel             | 1  | 20.5761  |
|    MegatronBertForQuestionAnswering     | 1  | 20.2754  |
|      GPT2ForSequenceClassification      | 1  | 19.0535  |
|            YituTechConvBert             | 1  | 17.6785  |
|           PegasusForCausalLM            | 1  | 16.5059  |
| BlenderbotSmallForConditionalGeneration | 1  | 16.2353  |
|     PLBartForConditionalGeneration      | 1  | 15.9034  |
|                CamemBert                | 1  | 15.6886  |
|            MBartForCausalLM             | 1  | 15.5201  |
|           LayoutLMForMaskedLM           | 1  | 15.3705  |
|       AlbertForQuestionAnswering        | 1  | 14.7732  |
|             OPTForCausalLM              | 1  |  14.347  |
|           ElectraForCausalLM            | 1  | 13.7837  |
|             BertForMaskedLM             | 1  | 13.7694  |
|           RobertaForCausalLM            | 1  | 13.7615  |
|            TrOCRForCausalLM             | 1  | 13.5365  |
|       RobertaForQuestionAnswering       | 1  | 13.5005  |
|       ElectraForQuestionAnswering       | 1  | 13.4956  |
|        BertForQuestionAnswering         | 1  | 13.3995  |
|               DistillGPT2               | 1  | 13.3664  |
|               GoogleFnet                | 1  | 12.4917  |
|         Speech2Text2ForCausalLM         | 1  | 12.3276  |
|            AlbertForMaskedLM            | 1  | 11.6542  |
|            PLBartForCausalLM            | 1  | 11.6237  |
|          DistilBertForMaskedLM          | 1  |  11.446  |
|     DistilBertForQuestionAnswering      | 1  | 11.2391  |
+-----------------------------------------+----+----------+

Peak Memory Compression Ratio

+-----------------------------------------+----+----------+
|                  name                   | bs | inductor |
+-----------------------------------------+----+----------+
|            AlbertForMaskedLM            | 1  |  2.802   |
|       AlbertForQuestionAnswering        | 1  |  2.583   |
|      BartForConditionalGeneration       | 1  |  2.1312  |
|       T5ForConditionalGeneration        | 1  |  2.1166  |
|                 BigBird                 | 1  |  2.0647  |
|                 T5Small                 | 1  |  1.9002  |
|      GPT2ForSequenceClassification      | 1  |  1.846   |
|          AllenaiLongformerBase          | 1  |  1.7207  |
|             BartForCausalLM             | 1  |  1.4786  |
|           DebertaForMaskedLM            | 1  |  1.4048  |
|               GoogleFnet                | 1  |  1.3821  |
|            XLNetLMHeadModel             | 1  |  1.3705  |
|       DebertaForQuestionAnswering       | 1  |  1.3687  |
|                CamemBert                | 1  |  1.3494  |
|            YituTechConvBert             | 1  |  1.3208  |
|           LayoutLMForMaskedLM           | 1  |  1.3173  |
|    LayoutLMForSequenceClassification    | 1  |  1.291   |
|           ElectraForCausalLM            | 1  |  1.2799  |
|               DistillGPT2               | 1  |  1.223   |
|       ElectraForQuestionAnswering       | 1  |  1.1931  |
|       MT5ForConditionalGeneration       | 1  |  1.1079  |
| BlenderbotSmallForConditionalGeneration | 1  |  1.1071  |
|     PegasusForConditionalGeneration     | 1  |  1.0873  |
|     M2M100ForConditionalGeneration      | 1  |  1.086   |
|      MBartForConditionalGeneration      | 1  |  1.0757  |
|         MegatronBertForCausalLM         | 1  |  1.0552  |
|            TrOCRForCausalLM             | 1  |  1.0532  |
|     PLBartForConditionalGeneration      | 1  |  1.051   |
|    MegatronBertForQuestionAnswering     | 1  |  1.0509  |
|             BertForMaskedLM             | 1  |  1.0492  |
|           RobertaForCausalLM            | 1  |  1.0466  |
|             XGLMForCausalLM             | 1  |  1.0427  |
|            PLBartForCausalLM            | 1  |  1.0404  |
|          DistilBertForMaskedLM          | 1  |  1.0374  |
|       BlenderbotSmallForCausalLM        | 1  |  1.037   |
|       RobertaForQuestionAnswering       | 1  |  1.0349  |
|        BertForQuestionAnswering         | 1  |  1.0348  |
|             OPTForCausalLM              | 1  |  1.0214  |
|     DistilBertForQuestionAnswering      | 1  |  1.0185  |
|            MBartForCausalLM             | 1  |  1.0161  |
|           PegasusForCausalLM            | 1  |  1.0117  |
|          MobileBertForMaskedLM          | 1  |  0.9992  |
|         Speech2Text2ForCausalLM         | 1  |  0.9802  |
|     MobileBertForQuestionAnswering      | 1  |  0.9763  |
+-----------------------------------------+----+----------+

timm_models suite with float32 precision

see more

Performance speedup

+---------------------------------+----+----------+
|              name               | bs | inductor |
+---------------------------------+----+----------+
|          pnasnet5large          | 1  |  1.7341  |
|        ese_vovnet19b_dw         | 1  |  1.3548  |
|           regnety_002           | 1  |  1.3477  |
|            lcnet_050            | 1  |  1.3356  |
|          inception_v3           | 1  |  1.3331  |
|          spnasnet_100           | 1  |  1.3173  |
|       gluon_inception_v3        | 1  |  1.313   |
|        adv_inception_v3         | 1  |  1.3073  |
|           mnasnet_100           | 1  |  1.2884  |
|          gmixer_24_224          | 1  |  1.2732  |
|           fbnetc_100            | 1  |  1.2512  |
|        gluon_xception65         | 1  |  1.2378  |
|         mobilenetv2_100         | 1  |  1.2232  |
|           res2next50            | 1  |  1.2148  |
|        res2net50_14w_8s         | 1  |  1.195   |
|          ghostnet_100           | 1  |  1.1809  |
|      mobilenetv3_large_100      | 1  |  1.169   |
|            fbnetv3_b            | 1  |  1.162   |
|             dla102              | 1  |  1.1449  |
|        res2net101_26w_4s        | 1  |  1.1322  |
|            hrnet_w18            | 1  |  1.1296  |
|            gernet_l             | 1  |  1.0825  |
|           selecsls42b           | 1  |  1.0775  |
|             dpn107              | 1  |  1.067   |
|          cspdarknet53           | 1  |  1.0659  |
|            repvgg_a2            | 1  |  1.0495  |
|           resnest101e           | 1  |  1.0373  |
|     swsl_resnext101_32x16d      | 1  |  1.0226  |
|            nfnet_l0             | 1  |  0.9919  |
|           dm_nfnet_f0           | 1  |  0.9164  |
|         visformer_small         | 1  |  0.9081  |
|      xcit_large_24_p8_224       | 1  |  0.8928  |
|      beit_base_patch16_224      | 1  |  0.8315  |
|  swin_base_patch4_window7_224   | 1  |  0.7798  |
| deit_base_distilled_patch16_224 | 1  |  0.7748  |
|           convit_base           | 1  |  0.7729  |
|      vit_base_patch16_224       | 1  |  0.7706  |
|          cait_m36_384           | 1  |  0.7586  |
|           volo_d1_224           | 1  |  0.7516  |
|         poolformer_m36          | 1  |  0.7361  |
|           tf_mixnet_l           | 1  |  0.7334  |
|            mixnet_l             | 1  |  0.7243  |
|          mixer_b16_224          | 1  |  0.718   |
|           rexnet_100            | 1  |  0.6964  |
|          resmlp_12_224          | 1  |  0.6721  |
|            pit_b_224            | 1  |  0.6687  |
|          convnext_base          | 1  |  0.6642  |
|        tnt_s_patch16_224        | 1  |  0.6411  |
|          jx_nest_base           | 1  |  0.6265  |
|        twins_pcpvt_base         | 1  |  0.6247  |
|         crossvit_9_240          | 1  |  0.6108  |
|           mobilevit_s           | 1  |  0.5634  |
|         coat_lite_mini          | 1  |  0.5461  |
|            tinynet_a            | 1  |  0.5407  |
|       tf_efficientnet_b0        | 1  |  0.4227  |
|          gmlp_s16_224           | 1  |  0.3766  |
|        eca_halonext26ts         | 0  |   0.0    |
|       eca_botnext26ts_256       | 0  |   0.0    |
|        sebotnet33ts_256         | 0  |   0.0    |
|          botnet26t_256          | 0  |   0.0    |
|        convmixer_768_32         | 0  |   0.0    |
+---------------------------------+----+----------+

Accuracy

+---------------------------------+----+---------------+
|              name               | bs |   inductor    |
+---------------------------------+----+---------------+
|        adv_inception_v3         | 1  |     pass      |
|             dpn107              | 1  |     pass      |
|      beit_base_patch16_224      | 1  |     pass      |
|          mixer_b16_224          | 1  |     pass      |
|        ese_vovnet19b_dw         | 1  |     pass      |
|         coat_lite_mini          | 1  |     pass      |
|           convit_base           | 1  |     pass      |
|          convnext_base          | 1  |     pass      |
|         crossvit_9_240          | 1  |     pass      |
|          cspdarknet53           | 1  |     pass      |
| deit_base_distilled_patch16_224 | 1  |     pass      |
|             dla102              | 1  |     pass      |
|           dm_nfnet_f0           | 1  |     pass      |
|            lcnet_050            | 1  |     pass      |
|           volo_d1_224           | 1  |     pass      |
|      xcit_large_24_p8_224       | 1  |     pass      |
|           fbnetc_100            | 1  |     pass      |
|        gluon_xception65         | 1  |     pass      |
|          jx_nest_base           | 1  |     pass      |
|          inception_v3           | 1  |     pass      |
|            hrnet_w18            | 1  |     pass      |
|          gmlp_s16_224           | 1  |     pass      |
|          gmixer_24_224          | 1  |     pass      |
|       gluon_inception_v3        | 1  |     pass      |
|            gernet_l             | 1  |     pass      |
|            fbnetv3_b            | 1  |     pass      |
|           mnasnet_100           | 1  |     pass      |
|            mixnet_l             | 1  |     pass      |
|      vit_base_patch16_224       | 1  |     pass      |
|           res2next50            | 1  |     pass      |
|         mobilenetv2_100         | 1  |     pass      |
|      mobilenetv3_large_100      | 1  |     pass      |
|           mobilevit_s           | 1  |     pass      |
|            nfnet_l0             | 1  |     pass      |
|            pit_b_224            | 1  |     pass      |
|          pnasnet5large          | 1  |     pass      |
|         poolformer_m36          | 1  |     pass      |
|           regnety_002           | 1  |     pass      |
|            repvgg_a2            | 1  |     pass      |
|         visformer_small         | 1  |     pass      |
|        res2net50_14w_8s         | 1  |     pass      |
|        res2net101_26w_4s        | 1  |     pass      |
|          resmlp_12_224          | 1  |     pass      |
|       tf_efficientnet_b0        | 1  |     pass      |
|        twins_pcpvt_base         | 1  |     pass      |
|        tnt_s_patch16_224        | 1  |     pass      |
|           resnest101e           | 1  |     pass      |
|           tf_mixnet_l           | 1  |     pass      |
|            tinynet_a            | 1  |     pass      |
|     swsl_resnext101_32x16d      | 1  |     pass      |
|  swin_base_patch4_window7_224   | 1  |     pass      |
|          spnasnet_100           | 1  |     pass      |
|           selecsls42b           | 1  |     pass      |
|           rexnet_100            | 1  |     pass      |
|        eca_halonext26ts         | 1  |  fail_to_run  |
|        convmixer_768_32         | 1  |  fail_to_run  |
|        sebotnet33ts_256         | 1  |  fail_to_run  |
|          botnet26t_256          | 1  |  fail_to_run  |
|       eca_botnext26ts_256       | 1  |  fail_to_run  |
|          ghostnet_100           | 1  | fail_accuracy |
|          cait_m36_384           | 1  | fail_accuracy |
+---------------------------------+----+---------------+

Compilation latency (sec)

+---------------------------------+----+----------+
|              name               | bs | inductor |
+---------------------------------+----+----------+
|          pnasnet5large          | 1  | 41.7089  |
|            hrnet_w18            | 1  | 39.2931  |
|           tf_mixnet_l           | 1  | 33.1455  |
|           rexnet_100            | 1  | 31.1758  |
|        twins_pcpvt_base         | 1  |  31.058  |
|          ghostnet_100           | 1  | 30.5595  |
|        res2net50_14w_8s         | 1  | 30.4734  |
|  swin_base_patch4_window7_224   | 1  |  30.162  |
|           mobilevit_s           | 1  | 29.9955  |
|         coat_lite_mini          | 1  | 29.8229  |
|            nfnet_l0             | 1  | 29.7563  |
|            mixnet_l             | 1  | 29.7319  |
|          cait_m36_384           | 1  | 29.3847  |
|           resnest101e           | 1  | 28.1108  |
|        res2net101_26w_4s        | 1  | 27.3083  |
|            fbnetv3_b            | 1  | 26.8071  |
|      xcit_large_24_p8_224       | 1  | 26.5514  |
|             dpn107              | 1  | 26.4295  |
|           dm_nfnet_f0           | 1  | 25.5659  |
|       tf_efficientnet_b0        | 1  | 24.8257  |
|         poolformer_m36          | 1  | 24.7907  |
|            tinynet_a            | 1  | 24.6314  |
|          jx_nest_base           | 1  | 22.7314  |
|           res2next50            | 1  | 22.3748  |
|      mobilenetv3_large_100      | 1  | 21.9447  |
|         crossvit_9_240          | 1  | 21.3376  |
|        gluon_xception65         | 1  | 21.1032  |
|        tnt_s_patch16_224        | 1  | 21.0291  |
|           volo_d1_224           | 1  | 20.9827  |
|       gluon_inception_v3        | 1  | 20.4352  |
|             dla102              | 1  | 19.9123  |
|        adv_inception_v3         | 1  | 19.8067  |
|          inception_v3           | 1  | 19.6737  |
|          convnext_base          | 1  | 19.1797  |
|          cspdarknet53           | 1  | 19.1125  |
|           fbnetc_100            | 1  | 19.0881  |
|          spnasnet_100           | 1  | 18.6126  |
|            pit_b_224            | 1  | 18.0546  |
|           regnety_002           | 1  | 17.5177  |
|     swsl_resnext101_32x16d      | 1  | 17.4402  |
|          gmlp_s16_224           | 1  | 17.4338  |
|         mobilenetv2_100         | 1  | 16.8956  |
|           mnasnet_100           | 1  | 16.8593  |
|           convit_base           | 1  | 16.6751  |
|         visformer_small         | 1  | 16.2995  |
|            gernet_l             | 1  | 14.7961  |
|          gmixer_24_224          | 1  | 14.7167  |
|           selecsls42b           | 1  | 13.9762  |
|        ese_vovnet19b_dw         | 1  | 13.5153  |
|          mixer_b16_224          | 1  | 13.4293  |
|            repvgg_a2            | 1  | 13.3048  |
| deit_base_distilled_patch16_224 | 1  | 12.8625  |
|            lcnet_050            | 1  |  12.858  |
|      beit_base_patch16_224      | 1  |  12.767  |
|      vit_base_patch16_224       | 1  | 12.5794  |
|          resmlp_12_224          | 1  | 10.5512  |
|          botnet26t_256          | 0  |   nan    |
|        convmixer_768_32         | 0  |   nan    |
|       eca_botnext26ts_256       | 0  |   nan    |
|        eca_halonext26ts         | 0  |   nan    |
|        sebotnet33ts_256         | 0  |   nan    |
+---------------------------------+----+----------+

Peak Memory Compression Ratio

+---------------------------------+----+----------+
|              name               | bs | inductor |
+---------------------------------+----+----------+
|      xcit_large_24_p8_224       | 1  |  2.3088  |
|          cait_m36_384           | 1  |  2.1772  |
|          pnasnet5large          | 1  |  1.6668  |
|        gluon_xception65         | 1  |  1.4668  |
|           mobilevit_s           | 1  |  1.2066  |
|           dm_nfnet_f0           | 1  |  1.205   |
|          jx_nest_base           | 1  |  1.1979  |
|     swsl_resnext101_32x16d      | 1  |  1.1881  |
|            nfnet_l0             | 1  |  1.1865  |
|           resnest101e           | 1  |  1.1833  |
|             dpn107              | 1  |  1.1777  |
|           convit_base           | 1  |  1.1426  |
|          cspdarknet53           | 1  |  1.1421  |
|             dla102              | 1  |  1.1357  |
|          convnext_base          | 1  |  1.1312  |
|         poolformer_m36          | 1  |  1.1191  |
|          gmixer_24_224          | 1  |  1.1105  |
|           selecsls42b           | 1  |  1.1089  |
|  swin_base_patch4_window7_224   | 1  |  1.1085  |
|            pit_b_224            | 1  |  1.1047  |
|           volo_d1_224           | 1  |  1.0932  |
|           res2next50            | 1  |  1.0922  |
|        res2net101_26w_4s        | 1  |   1.09   |
|        tnt_s_patch16_224        | 1  |  1.0877  |
| deit_base_distilled_patch16_224 | 1  |  1.0821  |
|      vit_base_patch16_224       | 1  |  1.0819  |
|          mixer_b16_224          | 1  |  1.0805  |
|        twins_pcpvt_base         | 1  |  1.0702  |
|           rexnet_100            | 1  |  1.0687  |
|       tf_efficientnet_b0        | 1  |  1.0654  |
|         mobilenetv2_100         | 1  |  1.0622  |
|           tf_mixnet_l           | 1  |  1.0591  |
|          inception_v3           | 1  |  1.0579  |
|        ese_vovnet19b_dw         | 1  |  1.0563  |
|       gluon_inception_v3        | 1  |  1.054   |
|        res2net50_14w_8s         | 1  |  1.0513  |
|            mixnet_l             | 1  |  1.0512  |
|          resmlp_12_224          | 1  |  1.0472  |
|        adv_inception_v3         | 1  |  1.0458  |
|         coat_lite_mini          | 1  |  1.0436  |
|            fbnetv3_b            | 1  |  1.0302  |
|            tinynet_a            | 1  |  1.0243  |
|            gernet_l             | 1  |  1.0241  |
|            repvgg_a2            | 1  |   1.02   |
|          spnasnet_100           | 1  |  1.0174  |
|           mnasnet_100           | 1  |  1.0144  |
|           fbnetc_100            | 1  |  1.0034  |
|            hrnet_w18            | 1  |  0.984   |
|         visformer_small         | 1  |  0.9648  |
|            lcnet_050            | 1  |  0.9643  |
|           regnety_002           | 1  |  0.9594  |
|      beit_base_patch16_224      | 1  |  0.9559  |
|          ghostnet_100           | 1  |  0.9436  |
|      mobilenetv3_large_100      | 1  |  0.8822  |
|         crossvit_9_240          | 1  |  0.8758  |
|          gmlp_s16_224           | 1  |  0.8741  |
|          botnet26t_256          | 0  |   nan    |
|        convmixer_768_32         | 0  |   nan    |
|       eca_botnext26ts_256       | 0  |   nan    |
|        eca_halonext26ts         | 0  |   nan    |
|        sebotnet33ts_256         | 0  |   nan    |
+---------------------------------+----+----------+

[yudongsi]

Performance Dashboard for float32 precision -- Single-Socket Multi-threads

Executive Summary

see more

We evaluate different backends across three benchmark suites - torchbench, huggingface and timm. We run these experiments on Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz. Each experiment runs one iteration of forward pass. For accuracy, we check the numerical correctness of forward pass outputs by comparing with native pytorch. We measure speedup by normalizing against the performance of native pytorch. We report mean compilation latency numbers and peak memory footprint reduction ratio.

Caveats

1. Batch size has been reduced to workaround OOM errors. Work is in progress to reduce peak memory footprint.
2. Experiments do not cover dynamic shapes.
3. Experimental setup does not have optimizer.

To measure performance, compilation latency and memory footprint reduction, we remove the models that fail accuracy checks.

Passrate

+----------+------------+-------------+-------------+
| Compiler | torchbench | huggingface | timm_models |
+----------+------------+-------------+-------------+
| inductor | 91%, 49/54 | 100%, 44/44 | 89%, 54/61  |
+----------+------------+-------------+-------------+

Geometric mean speedup

+----------+------------+-------------+-------------+
| Compiler | torchbench | huggingface | timm_models |
+----------+------------+-------------+-------------+
| inductor |   1.07x    |    1.07x    |    1.09x    |
+----------+------------+-------------+-------------+

Mean compilation time (seconds)

+----------+------------+-------------+-------------+
| Compiler | torchbench | huggingface | timm_models |
+----------+------------+-------------+-------------+
| inductor |    11.68    |    14.90    |    18.76    |
+----------+------------+-------------+-------------+

Peak memory footprint compression ratio (higher is better)

+----------+------------+-------------+-------------+
| Compiler | torchbench | huggingface | timm_models |
+----------+------------+-------------+-------------+
| inductor |   0.94x    |    0.96x    |    0.99x    |
+----------+------------+-------------+-------------+

torchbench suite with float32 precision

see more

Performance speedup

+-----------------------------------+------+----------+
|               name                |  bs  | inductor |
+-----------------------------------+------+----------+
|        shufflenet_v2_x1_0         |  64  |  1.8229  |
|            densenet121            |  64  |  1.4481  |
|            mnasnet1_0             |  32  |  1.2714  |
|             resnet18              |  8   |  1.2227  |
|           pytorch_unet            |  1   |  1.187   |
|           timm_resnest            |  32  |  1.1784  |
|           squeezenet1_1           |  16  |  1.1622  |
|       functorch_dp_cifar10        |  64  |  1.1606  |
|              alexnet              | 128  |  1.1507  |
|           mobilenet_v2            |  16  |  1.145   |
|        mobilenet_v3_large         |  32  |  1.1357  |
|            timm_vovnet            |  32  |  1.1106  |
|                drq                |  1   |  1.1049  |
|             resnet50              |  32  |  1.093   |
|               dcgan               | 256  |  1.0908  |
|   pytorch_CycleGAN_and_pix2pix    |  1   |  1.0891  |
|               vgg16               |  4   |  1.0725  |
|            hf_Reformer            |  1   |  1.0698  |
|            Super_SloMo            |  6   |  1.0689  |
|            hf_T5_large            |  1   |  1.0617  |
|          LearningToPaint          |  96  |  1.0579  |
|               dlrm                | 2048 |  1.0565  |
|        Background_Matting         |  1   |  1.0554  |
|            timm_regnet            |  32  |  1.0524  |
|            hf_BigBird             |  1   |  1.0321  |
|          pytorch_stargan          |  16  |  1.0292  |
|               hf_T5               |  1   |  1.0188  |
|    mobilenet_v2_quantized_qat     |  96  |  0.9988  |
|              demucs               |  1   |  0.9975  |
|            tts_angular            |  64  |  0.9962  |
|      resnet50_quantized_qat       |  32  |  0.9959  |
|           BERT_pytorch            |  2   |  0.9875  |
|           hf_GPT2_large           |  1   |  0.9401  |
|              hf_GPT2              |  1   |  0.9269  |
|           hf_Longformer           |  1   |  0.921   |
|             hf_Albert             |  1   |  0.8995  |
|              yolov3               |  8   |  0.8824  |
|            hf_T5_base             |  1   |  0.8824  |
|   timm_vision_transformer_large   |  8   |  0.8753  |
|          resnext50_32x4d          |  8   |  0.8594  |
|      nvidia_deeprecommender       | 256  |  0.8387  |
|           hf_DistilBert           |  1   |  0.8317  |
|            timm_nfnet             | 128  |  0.8173  |
|              hf_Bert              |  1   |  0.8049  |
|         timm_efficientnet         |  64  |  0.7993  |
|              hf_Bart              |  1   |  0.7981  |
| attention_is_all_you_need_pytorch |  32  |  0.728   |
|      timm_vision_transformer      |  8   |  0.637   |
|           lennard_jones           | 1000 |  0.3592  |
|             tacotron2             |  0   |   0.0    |
|        speech_transformer         |  0   |   0.0    |
|         soft_actor_critic         |  0   |   0.0    |
|           fastNLP_Bert            |  0   |   0.0    |
|         timm_efficientdet         |  0   |   0.0    |
+-----------------------------------+------+----------+

Accuracy

+-----------------------------------+----+------------------+
|               name                | bs |     inductor     |
+-----------------------------------+----+------------------+
|            hf_T5_large            | 2  | pass_due_to_skip |
|           hf_GPT2_large           | 2  | pass_due_to_skip |
|   timm_vision_transformer_large   | 2  | pass_due_to_skip |
|               hf_T5               | 2  |       pass       |
|            hf_Reformer            | 2  |       pass       |
|            hf_T5_base             | 2  |       pass       |
|          LearningToPaint          | 2  |       pass       |
|            Super_SloMo            | 2  |       pass       |
|              alexnet              | 2  |       pass       |
| attention_is_all_you_need_pytorch | 2  |       pass       |
|               dcgan               | 2  |       pass       |
|              demucs               | 1  |       pass       |
|            densenet121            | 2  |       pass       |
|               dlrm                | 2  |       pass       |
|                drq                | 1  |       pass       |
|              yolov3               | 2  |       pass       |
|           mobilenet_v2            | 2  |       pass       |
|             hf_Albert             | 2  |       pass       |
|              hf_Bart              | 2  |       pass       |
|              hf_Bert              | 2  |       pass       |
|            hf_BigBird             | 2  |       pass       |
|           hf_DistilBert           | 2  |       pass       |
|              hf_GPT2              | 2  |       pass       |
|           hf_Longformer           | 2  |       pass       |
|       functorch_dp_cifar10        | 2  |       pass       |
|           lennard_jones           | 2  |       pass       |
|        Background_Matting         | 1  |       pass       |
|          resnext50_32x4d          | 2  |       pass       |
|           BERT_pytorch            | 2  |       pass       |
|      resnet50_quantized_qat       | 2  |       pass       |
|    mobilenet_v2_quantized_qat     | 2  |       pass       |
|        mobilenet_v3_large         | 2  |       pass       |
|      nvidia_deeprecommender       | 2  |       pass       |
|   pytorch_CycleGAN_and_pix2pix    | 1  |       pass       |
|          pytorch_stargan          | 16 |       pass       |
|           pytorch_unet            | 2  |       pass       |
|               vgg16               | 2  |       pass       |
|             resnet50              | 2  |       pass       |
|             resnet18              | 2  |       pass       |
|            mnasnet1_0             | 2  |       pass       |
|        shufflenet_v2_x1_0         | 2  |       pass       |
|           squeezenet1_1           | 2  |       pass       |
|         timm_efficientnet         | 2  |       pass       |
|            timm_nfnet             | 2  |       pass       |
|            timm_regnet            | 2  |       pass       |
|           timm_resnest            | 2  |       pass       |
|      timm_vision_transformer      | 2  |       pass       |
|            timm_vovnet            | 2  |       pass       |
|            tts_angular            | 2  |       pass       |
|         soft_actor_critic         | 0  |      0.0000      |
|           fastNLP_Bert            | 0  |      0.0000      |
|             tacotron2             | 0  |      0.0000      |
|         timm_efficientdet         | 0  |      0.0000      |
|        speech_transformer         | 0  |      0.0000      |
+-----------------------------------+----+------------------+

Compilation latency (sec)

+-----------------------------------+------+----------+
|               name                |  bs  | inductor |
+-----------------------------------+------+----------+
|            hf_T5_large            |  1   | 33.7788  |
|   timm_vision_transformer_large   |  8   |  32.45   |
|            timm_nfnet             | 128  | 23.7158  |
|           hf_GPT2_large           |  1   | 22.4235  |
|            densenet121            |  64  | 21.8843  |
|            hf_T5_base             |  1   | 21.4543  |
|            hf_BigBird             |  1   | 21.3205  |
|           hf_Longformer           |  1   | 20.3187  |
|              yolov3               |  8   | 18.3641  |
|            Super_SloMo            |  6   | 16.8824  |
|         timm_efficientnet         |  64  | 14.9878  |
|            timm_regnet            |  32  | 14.5997  |
|              hf_Bert              |  1   | 13.6024  |
|              hf_Bart              |  1   | 12.6615  |
|            hf_Reformer            |  1   | 12.1682  |
| attention_is_all_you_need_pytorch |  32  | 11.8647  |
|               hf_T5               |  1   | 11.8495  |
|           BERT_pytorch            |  2   | 11.4679  |
|              hf_GPT2              |  1   | 10.6672  |
|      timm_vision_transformer      |  8   | 10.5754  |
|        Background_Matting         |  1   | 10.4842  |
|             resnet50              |  32  | 10.3343  |
|             hf_Albert             |  1   | 10.2016  |
|            timm_vovnet            |  32  | 10.1441  |
|        mobilenet_v3_large         |  32  | 10.0769  |
|        shufflenet_v2_x1_0         |  64  |  9.7873  |
|          resnext50_32x4d          |  8   |  9.7049  |
|            mnasnet1_0             |  32  |  9.2318  |
|           mobilenet_v2            |  16  |  9.2068  |
|           hf_DistilBert           |  1   |  8.9652  |
|           timm_resnest            |  32  |  8.827   |
|           pytorch_unet            |  1   |  8.7658  |
|       functorch_dp_cifar10        |  64  |  8.4817  |
|   pytorch_CycleGAN_and_pix2pix    |  1   |  8.1832  |
|          pytorch_stargan          |  16  |  8.176   |
|          LearningToPaint          |  96  |  7.7343  |
|           squeezenet1_1           |  16  |  7.5185  |
|             resnet18              |  8   |  7.5156  |
|      nvidia_deeprecommender       | 256  |  7.5014  |
|               vgg16               |  4   |  7.4072  |
|               dlrm                | 2048 |  7.3599  |
|              alexnet              | 128  |  7.1185  |
|            tts_angular            |  64  |  7.072   |
|                drq                |  1   |  6.7346  |
|           lennard_jones           | 1000 |  6.6819  |
|              demucs               |  1   |  1.4613  |
|               dcgan               | 256  |  0.2598  |
|    mobilenet_v2_quantized_qat     |  96  |  0.2521  |
|      resnet50_quantized_qat       |  32  |  0.2215  |
|           fastNLP_Bert            |  0   |   nan    |
|         soft_actor_critic         |  0   |   nan    |
|        speech_transformer         |  0   |   nan    |
|             tacotron2             |  0   |   nan    |
|         timm_efficientdet         |  0   |   nan    |
+-----------------------------------+------+----------+

Peak Memory Compression Ratio

+-----------------------------------+------+----------+
|               name                |  bs  | inductor |
+-----------------------------------+------+----------+
|              demucs               |  1   |  0.9988  |
|               dlrm                | 2048 |  0.9968  |
|      resnet50_quantized_qat       |  32  |  0.9959  |
|        Background_Matting         |  1   |  0.994   |
|   timm_vision_transformer_large   |  8   |  0.9935  |
|         timm_efficientnet         |  64  |  0.9925  |
|               vgg16               |  4   |  0.9924  |
|            timm_nfnet             | 128  |  0.9918  |
|          LearningToPaint          |  96  |  0.9915  |
|           pytorch_unet            |  1   |  0.9909  |
|            timm_regnet            |  32  |  0.9908  |
|            hf_BigBird             |  1   |  0.9908  |
|            Super_SloMo            |  6   |  0.9902  |
|            densenet121            |  64  |  0.9894  |
|             resnet50              |  32  |  0.9881  |
|            mnasnet1_0             |  32  |  0.9859  |
|                drq                |  1   |  0.9854  |
|            hf_T5_base             |  1   |  0.9838  |
|           lennard_jones           | 1000 |  0.9822  |
|        mobilenet_v3_large         |  32  |  0.9817  |
| attention_is_all_you_need_pytorch |  32  |  0.9815  |
|           mobilenet_v2            |  16  |  0.9805  |
|           hf_GPT2_large           |  1   |  0.9805  |
|        shufflenet_v2_x1_0         |  64  |  0.9804  |
|            timm_vovnet            |  32  |  0.9794  |
|            tts_angular            |  64  |  0.9786  |
|              alexnet              | 128  |  0.9784  |
|           hf_DistilBert           |  1   |  0.9774  |
|          resnext50_32x4d          |  8   |  0.9757  |
|    mobilenet_v2_quantized_qat     |  96  |  0.9735  |
|             hf_Albert             |  1   |  0.9723  |
|              hf_Bert              |  1   |  0.9689  |
|              hf_Bart              |  1   |  0.9658  |
|          pytorch_stargan          |  16  |  0.9632  |
|              hf_GPT2              |  1   |  0.9616  |
|   pytorch_CycleGAN_and_pix2pix    |  1   |  0.949   |
|      timm_vision_transformer      |  8   |  0.9335  |
|             resnet18              |  8   |  0.9325  |
|               dcgan               | 256  |  0.9308  |
|           hf_Longformer           |  1   |  0.9176  |
|           BERT_pytorch            |  2   |  0.9168  |
|            hf_Reformer            |  1   |  0.9164  |
|       functorch_dp_cifar10        |  64  |  0.9003  |
|           timm_resnest            |  32  |  0.8899  |
|              yolov3               |  8   |  0.8676  |
|               hf_T5               |  1   |  0.7678  |
|           squeezenet1_1           |  16  |  0.7248  |
|            hf_T5_large            |  1   |  0.6029  |
|      nvidia_deeprecommender       | 256  |  0.5832  |
|           fastNLP_Bert            |  0   |   nan    |
|         soft_actor_critic         |  0   |   nan    |
|        speech_transformer         |  0   |   nan    |
|             tacotron2             |  0   |   nan    |
|         timm_efficientdet         |  0   |   nan    |
+-----------------------------------+------+----------+

huggingface suite with float32 precision

see more

Performance speedup

+-----------------------------------------+-----+----------+
|                  name                   | bs  | inductor |
+-----------------------------------------+-----+----------+
|       MT5ForConditionalGeneration       |  8  |   2.52   |
|            XLNetLMHeadModel             | 32  |  1.9735  |
|     MobileBertForQuestionAnswering      | 64  |  1.3506  |
|               DistillGPT2               |  1  |  1.3222  |
|               GoogleFnet                |  1  |  1.2726  |
|            YituTechConvBert             |  1  |  1.1963  |
|          MobileBertForMaskedLM          | 32  |  1.1388  |
|                 BigBird                 |  1  |  1.0872  |
|     M2M100ForConditionalGeneration      |  8  |  1.0839  |
|             XGLMForCausalLM             |  8  |  1.0554  |
|                 T5Small                 |  1  |  1.0447  |
|          AllenaiLongformerBase          |  1  |  1.0342  |
|            AlbertForMaskedLM            |  4  |  1.0302  |
|       AlbertForQuestionAnswering        |  4  |  1.0284  |
|             OPTForCausalLM              | 32  |  1.0192  |
|           DebertaForMaskedLM            |  4  |  0.9714  |
|                CamemBert                |  1  |  0.9494  |
|      GPT2ForSequenceClassification      |  4  |  0.9432  |
|         Speech2Text2ForCausalLM         | 128 |  0.9414  |
|       DebertaForQuestionAnswering       |  8  |  0.9174  |
|           RobertaForCausalLM            | 64  |  0.8998  |
|     PLBartForConditionalGeneration      | 16  |  0.8968  |
|           ElectraForCausalLM            | 32  |  0.8924  |
|         MegatronBertForCausalLM         | 16  |   0.89   |
|      MBartForConditionalGeneration      | 16  |  0.8895  |
|            PLBartForCausalLM            | 32  |  0.8875  |
|     PegasusForConditionalGeneration     | 16  |  0.8874  |
|    LayoutLMForSequenceClassification    | 16  |  0.8763  |
|            TrOCRForCausalLM             | 32  |  0.8712  |
|       RobertaForQuestionAnswering       | 128 |  0.8694  |
|    MegatronBertForQuestionAnswering     | 16  |  0.8664  |
|           PegasusForCausalLM            | 32  |  0.8664  |
|           LayoutLMForMaskedLM           | 16  |  0.8655  |
|            MBartForCausalLM             | 32  |  0.8629  |
|       T5ForConditionalGeneration        |  4  |  0.8604  |
|             BertForMaskedLM             | 64  |  0.8416  |
|     DistilBertForQuestionAnswering      | 64  |  0.8389  |
|        BertForQuestionAnswering         | 128 |  0.836   |
|          DistilBertForMaskedLM          | 64  |  0.8349  |
|             BartForCausalLM             |  4  |  0.8196  |
|       ElectraForQuestionAnswering       | 64  |  0.8134  |
|      BartForConditionalGeneration       |  2  |  0.8084  |
| BlenderbotSmallForConditionalGeneration | 64  |  0.8081  |
|       BlenderbotSmallForCausalLM        | 64  |  0.793   |
+-----------------------------------------+-----+----------+

Accuracy

+-----------------------------------------+----+----------+
|                  name                   | bs | inductor |
+-----------------------------------------+----+----------+
|            AlbertForMaskedLM            | 1  |   pass   |
|       AlbertForQuestionAnswering        | 1  |   pass   |
|                CamemBert                | 1  |   pass   |
|          AllenaiLongformerBase          | 1  |   pass   |
|             BartForCausalLM             | 1  |   pass   |
|      BartForConditionalGeneration       | 1  |   pass   |
|             BertForMaskedLM             | 1  |   pass   |
|        BertForQuestionAnswering         | 1  |   pass   |
|                 BigBird                 | 1  |   pass   |
|       BlenderbotSmallForCausalLM        | 1  |   pass   |
| BlenderbotSmallForConditionalGeneration | 1  |   pass   |
|           DebertaForMaskedLM            | 1  |   pass   |
|           LayoutLMForMaskedLM           | 1  |   pass   |
|       DebertaForQuestionAnswering       | 1  |   pass   |
|          DistilBertForMaskedLM          | 1  |   pass   |
|     DistilBertForQuestionAnswering      | 1  |   pass   |
|               DistillGPT2               | 1  |   pass   |
|           ElectraForCausalLM            | 1  |   pass   |
|       ElectraForQuestionAnswering       | 1  |   pass   |
|      GPT2ForSequenceClassification      | 1  |   pass   |
|               GoogleFnet                | 1  |   pass   |
|    LayoutLMForSequenceClassification    | 1  |   pass   |
|     M2M100ForConditionalGeneration      | 1  |   pass   |
|            MBartForCausalLM             | 1  |   pass   |
|     PLBartForConditionalGeneration      | 1  |   pass   |
|      MBartForConditionalGeneration      | 1  |   pass   |
|       MT5ForConditionalGeneration       | 1  |   pass   |
|         MegatronBertForCausalLM         | 1  |   pass   |
|    MegatronBertForQuestionAnswering     | 1  |   pass   |
|          MobileBertForMaskedLM          | 1  |   pass   |
|     MobileBertForQuestionAnswering      | 1  |   pass   |
|             OPTForCausalLM              | 1  |   pass   |
|            PLBartForCausalLM            | 1  |   pass   |
|           PegasusForCausalLM            | 1  |   pass   |
|            XLNetLMHeadModel             | 1  |   pass   |
|     PegasusForConditionalGeneration     | 1  |   pass   |
|           RobertaForCausalLM            | 1  |   pass   |
|       RobertaForQuestionAnswering       | 1  |   pass   |
|         Speech2Text2ForCausalLM         | 1  |   pass   |
|       T5ForConditionalGeneration        | 1  |   pass   |
|                 T5Small                 | 1  |   pass   |
|            TrOCRForCausalLM             | 1  |   pass   |
|             XGLMForCausalLM             | 1  |   pass   |
|            YituTechConvBert             | 1  |   pass   |
+-----------------------------------------+----+----------+

Compilation latency (sec)

+-----------------------------------------+-----+----------+
|                  name                   | bs  | inductor |
+-----------------------------------------+-----+----------+
|     PegasusForConditionalGeneration     | 16  | 22.3478  |
|          MobileBertForMaskedLM          | 32  | 21.9429  |
|       DebertaForQuestionAnswering       |  8  |  21.77   |
|            YituTechConvBert             |  1  | 21.6894  |
|     MobileBertForQuestionAnswering      | 64  | 21.4578  |
|          AllenaiLongformerBase          |  1  | 21.2457  |
|           DebertaForMaskedLM            |  4  | 20.9841  |
|     M2M100ForConditionalGeneration      |  8  | 20.5287  |
|                 BigBird                 |  1  | 20.2965  |
|      BartForConditionalGeneration       |  2  | 19.7537  |
|      MBartForConditionalGeneration      | 16  | 19.0224  |
|             XGLMForCausalLM             |  8  | 18.5344  |
|         MegatronBertForCausalLM         | 16  | 17.2338  |
|    MegatronBertForQuestionAnswering     | 16  | 16.8028  |
|       RobertaForQuestionAnswering       | 128 | 16.3171  |
| BlenderbotSmallForConditionalGeneration | 64  | 15.7945  |
|           PegasusForCausalLM            | 32  |  14.928  |
|            AlbertForMaskedLM            |  4  | 14.4438  |
|       AlbertForQuestionAnswering        |  4  | 14.3569  |
|        BertForQuestionAnswering         | 128 | 14.2543  |
|       MT5ForConditionalGeneration       |  8  | 14.1973  |
|           LayoutLMForMaskedLM           | 16  | 13.6621  |
|       T5ForConditionalGeneration        |  4  | 13.6308  |
|             BartForCausalLM             |  4  | 13.3232  |
|       ElectraForQuestionAnswering       | 64  | 13.2929  |
|           RobertaForCausalLM            | 64  | 13.2914  |
|     PLBartForConditionalGeneration      | 16  | 13.2436  |
|           ElectraForCausalLM            | 32  | 13.2227  |
|             BertForMaskedLM             | 64  |  13.182  |
|    LayoutLMForSequenceClassification    | 16  | 13.1042  |
|            MBartForCausalLM             | 32  | 12.5802  |
|      GPT2ForSequenceClassification      |  4  | 12.4793  |
|            TrOCRForCausalLM             | 32  | 12.4106  |
|             OPTForCausalLM              | 32  | 12.0205  |
|                 T5Small                 |  1  | 11.7247  |
|                CamemBert                |  1  | 11.1804  |
|               GoogleFnet                |  1  | 11.1298  |
|         Speech2Text2ForCausalLM         | 128 | 10.7654  |
|       BlenderbotSmallForCausalLM        | 64  | 10.7171  |
|          DistilBertForMaskedLM          | 64  | 10.1725  |
|     DistilBertForQuestionAnswering      | 64  |  9.6337  |
|            PLBartForCausalLM            | 32  |  9.5557  |
|               DistillGPT2               |  1  |  8.4453  |
|            XLNetLMHeadModel             | 32  |  4.8687  |
+-----------------------------------------+-----+----------+

Peak Memory Compression Ratio

+-----------------------------------------+-----+----------+
|                  name                   | bs  | inductor |
+-----------------------------------------+-----+----------+
|            AlbertForMaskedLM            |  4  |  0.9978  |
|       AlbertForQuestionAnswering        |  4  |  0.9976  |
|       ElectraForQuestionAnswering       | 64  |  0.9962  |
|             BartForCausalLM             |  4  |  0.9959  |
|           RobertaForCausalLM            | 64  |  0.9956  |
|           ElectraForCausalLM            | 32  |  0.9954  |
|       BlenderbotSmallForCausalLM        | 64  |  0.9952  |
|             BertForMaskedLM             | 64  |  0.9951  |
| BlenderbotSmallForConditionalGeneration | 64  |  0.9947  |
|          DistilBertForMaskedLM          | 64  |  0.9944  |
|           PegasusForCausalLM            | 32  |  0.9944  |
|            TrOCRForCausalLM             | 32  |  0.9943  |
|      GPT2ForSequenceClassification      |  4  |  0.9943  |
|       DebertaForQuestionAnswering       |  8  |  0.9937  |
|            MBartForCausalLM             | 32  |  0.9935  |
|       T5ForConditionalGeneration        |  4  |  0.9931  |
|             OPTForCausalLM              | 32  |  0.993   |
|         Speech2Text2ForCausalLM         | 128 |  0.9929  |
|            PLBartForCausalLM            | 32  |  0.9924  |
|     DistilBertForQuestionAnswering      | 64  |  0.9916  |
|     PegasusForConditionalGeneration     | 16  |  0.9915  |
|                 BigBird                 |  1  |  0.9914  |
|        BertForQuestionAnswering         | 128 |  0.9913  |
|      BartForConditionalGeneration       |  2  |  0.9913  |
|       RobertaForQuestionAnswering       | 128 |  0.9912  |
|           DebertaForMaskedLM            |  4  |  0.9893  |
|           LayoutLMForMaskedLM           | 16  |  0.9876  |
|      MBartForConditionalGeneration      | 16  |  0.9874  |
|     PLBartForConditionalGeneration      | 16  |  0.987   |
|               GoogleFnet                |  1  |  0.987   |
|    LayoutLMForSequenceClassification    | 16  |  0.9853  |
|               DistillGPT2               |  1  |  0.9838  |
|            XLNetLMHeadModel             | 32  |  0.9773  |
|          MobileBertForMaskedLM          | 32  |  0.9771  |
|         MegatronBertForCausalLM         | 16  |  0.9645  |
|    MegatronBertForQuestionAnswering     | 16  |  0.9602  |
|     M2M100ForConditionalGeneration      |  8  |  0.9538  |
|             XGLMForCausalLM             |  8  |  0.9248  |
|          AllenaiLongformerBase          |  1  |  0.8733  |
|     MobileBertForQuestionAnswering      | 64  |  0.848   |
|                CamemBert                |  1  |  0.8381  |
|            YituTechConvBert             |  1  |  0.8042  |
|       MT5ForConditionalGeneration       |  8  |  0.7392  |
|                 T5Small                 |  1  |  0.7249  |
+-----------------------------------------+-----+----------+

timm_models suite with float32 precision

see more

Performance speedup

+---------------------------------+-----+----------+
|              name               | bs  | inductor |
+---------------------------------+-----+----------+
|          pnasnet5large          | 16  |  1.5267  |
|        gluon_xception65         | 32  |  1.4004  |
|           resnest101e           | 64  |  1.3192  |
|          inception_v3           | 128 |  1.3004  |
|        adv_inception_v3         | 128 |  1.2898  |
|       gluon_inception_v3        | 128 |  1.2825  |
|           fbnetc_100            | 128 |  1.2105  |
|           res2next50            | 128 |  1.2083  |
|        res2net50_14w_8s         | 128 |  1.2004  |
|        res2net101_26w_4s        | 64  |  1.1989  |
|           volo_d1_224           | 64  |   1.19   |
|          spnasnet_100           | 128 |  1.1888  |
|           mnasnet_100           | 128 |  1.1874  |
|      mobilenetv3_large_100      | 128 |  1.1814  |
|        ese_vovnet19b_dw         | 128 |  1.1771  |
|            hrnet_w18            | 128 |  1.1738  |
|          ghostnet_100           | 128 |  1.1673  |
|            lcnet_050            | 128 |  1.1595  |
|             dpn107              | 32  |  1.1524  |
|            gernet_l             | 128 |  1.1266  |
|            repvgg_a2            | 128 |  1.1169  |
|         visformer_small         | 128 |  1.1012  |
|            fbnetv3_b            | 128 |  1.1005  |
|           selecsls42b           | 128 |  1.0886  |
|         mobilenetv2_100         | 128 |  1.0687  |
|           regnety_002           | 128 |  1.0555  |
|           tf_mixnet_l           | 128 |  1.0453  |
|          cspdarknet53           | 64  |  1.0445  |
|      xcit_large_24_p8_224       |  5  |  1.0317  |
|          gmixer_24_224          | 128 |   1.03   |
|            mixnet_l             | 128 |   1.0    |
|          cait_m36_384           |  4  |  0.9731  |
|             dla102              | 128 |  0.9443  |
|            nfnet_l0             | 128 |  0.9023  |
|  swin_base_patch4_window7_224   | 64  |  0.8897  |
|           dm_nfnet_f0           | 128 |  0.885   |
|      beit_base_patch16_224      | 64  |  0.8841  |
|           rexnet_100            | 128 |  0.882   |
|           convit_base           | 64  |  0.8643  |
|          convnext_base          | 64  |  0.8637  |
| deit_base_distilled_patch16_224 | 64  |  0.8615  |
|       tf_efficientnet_b0        | 128 |  0.8598  |
|      vit_base_patch16_224       | 64  |  0.8571  |
|         poolformer_m36          | 64  |  0.8554  |
|          jx_nest_base           | 32  |  0.8342  |
|          resmlp_12_224          | 128 |  0.8261  |
|            tinynet_a            | 128 |  0.8205  |
|           mobilevit_s           | 64  |  0.8145  |
|        tnt_s_patch16_224        | 128 |  0.8144  |
|            pit_b_224            | 64  |  0.7913  |
|          mixer_b16_224          | 128 |  0.7906  |
|         coat_lite_mini          | 128 |  0.7535  |
|        twins_pcpvt_base         | 64  |  0.7371  |
|         crossvit_9_240          | 128 |  0.7225  |
|          gmlp_s16_224           | 128 |  0.5764  |
|     swsl_resnext101_32x16d      | 32  |  0.0717  |
|        convmixer_768_32         |  0  |   0.0    |
|        sebotnet33ts_256         |  0  |   0.0    |
|        eca_halonext26ts         |  0  |   0.0    |
|          botnet26t_256          |  0  |   0.0    |
|       eca_botnext26ts_256       |  0  |   0.0    |
+---------------------------------+-----+----------+

Accuracy

+---------------------------------+----+---------------+
|              name               | bs |   inductor    |
+---------------------------------+----+---------------+
|        adv_inception_v3         | 2  |     pass      |
|             dpn107              | 2  |     pass      |
|      beit_base_patch16_224      | 2  |     pass      |
|          mixer_b16_224          | 2  |     pass      |
|        ese_vovnet19b_dw         | 2  |     pass      |
|         coat_lite_mini          | 2  |     pass      |
|           convit_base           | 2  |     pass      |
|          convnext_base          | 2  |     pass      |
|         crossvit_9_240          | 2  |     pass      |
|          cspdarknet53           | 2  |     pass      |
| deit_base_distilled_patch16_224 | 2  |     pass      |
|             dla102              | 2  |     pass      |
|           dm_nfnet_f0           | 2  |     pass      |
|            lcnet_050            | 2  |     pass      |
|           volo_d1_224           | 2  |     pass      |
|      xcit_large_24_p8_224       | 2  |     pass      |
|           fbnetc_100            | 2  |     pass      |
|        gluon_xception65         | 2  |     pass      |
|          jx_nest_base           | 2  |     pass      |
|          inception_v3           | 2  |     pass      |
|            hrnet_w18            | 2  |     pass      |
|          gmlp_s16_224           | 2  |     pass      |
|          gmixer_24_224          | 2  |     pass      |
|       gluon_inception_v3        | 2  |     pass      |
|            gernet_l             | 2  |     pass      |
|            fbnetv3_b            | 2  |     pass      |
|           mnasnet_100           | 2  |     pass      |
|            mixnet_l             | 2  |     pass      |
|      vit_base_patch16_224       | 2  |     pass      |
|           res2next50            | 2  |     pass      |
|         mobilenetv2_100         | 2  |     pass      |
|      mobilenetv3_large_100      | 2  |     pass      |
|           mobilevit_s           | 2  |     pass      |
|            nfnet_l0             | 2  |     pass      |
|            pit_b_224            | 2  |     pass      |
|          pnasnet5large          | 2  |     pass      |
|         poolformer_m36          | 2  |     pass      |
|           regnety_002           | 2  |     pass      |
|            repvgg_a2            | 2  |     pass      |
|         visformer_small         | 2  |     pass      |
|        res2net50_14w_8s         | 2  |     pass      |
|        res2net101_26w_4s        | 2  |     pass      |
|          resmlp_12_224          | 2  |     pass      |
|       tf_efficientnet_b0        | 2  |     pass      |
|        twins_pcpvt_base         | 2  |     pass      |
|        tnt_s_patch16_224        | 2  |     pass      |
|           resnest101e           | 2  |     pass      |
|           tf_mixnet_l           | 2  |     pass      |
|            tinynet_a            | 2  |     pass      |
|     swsl_resnext101_32x16d      | 2  |     pass      |
|  swin_base_patch4_window7_224   | 2  |     pass      |
|          spnasnet_100           | 2  |     pass      |
|           selecsls42b           | 2  |     pass      |
|           rexnet_100            | 2  |     pass      |
|        eca_halonext26ts         | 2  |  fail_to_run  |
|        convmixer_768_32         | 2  |  fail_to_run  |
|        sebotnet33ts_256         | 2  |  fail_to_run  |
|          botnet26t_256          | 2  |  fail_to_run  |
|       eca_botnext26ts_256       | 2  |  fail_to_run  |
|          ghostnet_100           | 2  | fail_accuracy |
|          cait_m36_384           | 2  | fail_accuracy |
+---------------------------------+----+---------------+

Compilation latency (sec)

+---------------------------------+-----+----------+
|              name               | bs  | inductor |
+---------------------------------+-----+----------+
|     swsl_resnext101_32x16d      | 32  | 116.6718 |
|          pnasnet5large          | 16  |  32.273  |
|           dm_nfnet_f0           | 128 | 30.8393  |
|            hrnet_w18            | 128 | 28.1686  |
|        tnt_s_patch16_224        | 128 | 27.0522  |
|  swin_base_patch4_window7_224   | 64  | 26.1303  |
|          cait_m36_384           |  4  | 24.0501  |
|           tf_mixnet_l           | 128 |  23.045  |
|        twins_pcpvt_base         | 64  | 22.4356  |
|           mobilevit_s           | 64  | 21.9736  |
|         poolformer_m36          | 64  | 21.9105  |
|      xcit_large_24_p8_224       |  5  | 21.1167  |
|          inception_v3           | 128 | 21.0264  |
|            mixnet_l             | 128 | 20.9835  |
|        res2net50_14w_8s         | 128 | 20.5074  |
|           rexnet_100            | 128 | 19.8709  |
|             dla102              | 128 | 19.7566  |
|            fbnetv3_b            | 128 | 19.6552  |
|           resnest101e           | 64  | 19.5838  |
|            nfnet_l0             | 128 | 19.4525  |
|        res2net101_26w_4s        | 64  | 19.4293  |
|             dpn107              | 32  | 19.1965  |
|          jx_nest_base           | 32  |  18.167  |
|          gmlp_s16_224           | 128 | 16.9396  |
|          convnext_base          | 64  | 16.9235  |
|           volo_d1_224           | 64  | 16.5552  |
|         crossvit_9_240          | 128 | 16.5376  |
|       tf_efficientnet_b0        | 128 | 16.2456  |
|           convit_base           | 64  |  16.061  |
|           res2next50            | 128 | 15.5874  |
|         coat_lite_mini          | 128 | 15.5764  |
|            pit_b_224            | 64  | 15.2213  |
|        adv_inception_v3         | 128 | 14.4561  |
|       gluon_inception_v3        | 128 | 14.2199  |
|        gluon_xception65         | 32  | 14.1294  |
|          mixer_b16_224          | 128 | 14.0365  |
|          gmixer_24_224          | 128 | 13.8209  |
|            tinynet_a            | 128 |  13.23   |
|      beit_base_patch16_224      | 64  | 13.1765  |
|         visformer_small         | 128 | 13.0997  |
|      mobilenetv3_large_100      | 128 | 12.8485  |
| deit_base_distilled_patch16_224 | 64  |  12.723  |
|          cspdarknet53           | 64  | 12.7215  |
|      vit_base_patch16_224       | 64  |  12.632  |
|          ghostnet_100           | 128 | 11.9344  |
|            gernet_l             | 128 | 11.7819  |
|           fbnetc_100            | 128 |  11.327  |
|          spnasnet_100           | 128 | 11.2226  |
|           regnety_002           | 128 |  11.123  |
|            repvgg_a2            | 128 | 10.9643  |
|         mobilenetv2_100         | 128 | 10.8973  |
|        ese_vovnet19b_dw         | 128 | 10.5822  |
|           mnasnet_100           | 128 | 10.2721  |
|          resmlp_12_224          | 128 | 10.0726  |
|           selecsls42b           | 128 |  9.447   |
|            lcnet_050            | 128 |  9.1307  |
|          botnet26t_256          |  0  |   nan    |
|        convmixer_768_32         |  0  |   nan    |
|       eca_botnext26ts_256       |  0  |   nan    |
|        eca_halonext26ts         |  0  |   nan    |
|        sebotnet33ts_256         |  0  |   nan    |
+---------------------------------+-----+----------+

Peak Memory Compression Ratio

+---------------------------------+-----+----------+
|              name               | bs  | inductor |
+---------------------------------+-----+----------+
|        ese_vovnet19b_dw         | 128 |  0.9975  |
|          mixer_b16_224          | 128 |  0.9972  |
|        adv_inception_v3         | 128 |  0.9971  |
|           dm_nfnet_f0           | 128 |  0.9968  |
|           convit_base           | 64  |  0.9967  |
| deit_base_distilled_patch16_224 | 64  |  0.9963  |
|      vit_base_patch16_224       | 64  |  0.9963  |
|             dla102              | 128 |  0.9963  |
|          gmixer_24_224          | 128 |  0.9962  |
|          resmlp_12_224          | 128 |  0.9962  |
|       gluon_inception_v3        | 128 |  0.9962  |
|        gluon_xception65         | 32  |  0.9962  |
|          inception_v3           | 128 |  0.9961  |
|            fbnetv3_b            | 128 |  0.996   |
|        res2net50_14w_8s         | 128 |  0.996   |
|      beit_base_patch16_224      | 64  |  0.9959  |
|           mobilevit_s           | 64  |  0.9957  |
|          convnext_base          | 64  |  0.9957  |
|            gernet_l             | 128 |  0.9957  |
|           rexnet_100            | 128 |  0.9956  |
|         visformer_small         | 128 |  0.9954  |
|            pit_b_224            | 64  |  0.9953  |
|         coat_lite_mini          | 128 |  0.9952  |
|            hrnet_w18            | 128 |  0.9949  |
|        tnt_s_patch16_224        | 128 |  0.9949  |
|           res2next50            | 128 |  0.9947  |
|            nfnet_l0             | 128 |  0.9946  |
|            mixnet_l             | 128 |  0.9946  |
|           tf_mixnet_l           | 128 |  0.9943  |
|            repvgg_a2            | 128 |  0.994   |
|        res2net101_26w_4s        | 64  |  0.994   |
|       tf_efficientnet_b0        | 128 |  0.9938  |
|          cait_m36_384           |  4  |  0.9936  |
|           resnest101e           | 64  |  0.9936  |
|             dpn107              | 32  |  0.9935  |
|  swin_base_patch4_window7_224   | 64  |  0.9934  |
|         poolformer_m36          | 64  |  0.9933  |
|         crossvit_9_240          | 128 |  0.9919  |
|          pnasnet5large          | 16  |  0.9919  |
|          jx_nest_base           | 32  |  0.9918  |
|      mobilenetv3_large_100      | 128 |  0.9914  |
|        twins_pcpvt_base         | 64  |  0.9914  |
|           volo_d1_224           | 64  |  0.9907  |
|      xcit_large_24_p8_224       |  5  |  0.9902  |
|     swsl_resnext101_32x16d      | 32  |  0.989   |
|            lcnet_050            | 128 |  0.9886  |
|           regnety_002           | 128 |  0.9871  |
|           fbnetc_100            | 128 |  0.9865  |
|           selecsls42b           | 128 |  0.9864  |
|          cspdarknet53           | 64  |  0.9863  |
|           mnasnet_100           | 128 |  0.9839  |
|          spnasnet_100           | 128 |  0.9833  |
|          ghostnet_100           | 128 |  0.9796  |
|            tinynet_a            | 128 |  0.9731  |
|          gmlp_s16_224           | 128 |  0.9425  |
|         mobilenetv2_100         | 128 |  0.8072  |
|          botnet26t_256          |  0  |   nan    |
|        convmixer_768_32         |  0  |   nan    |
|       eca_botnext26ts_256       |  0  |   nan    |
|        eca_halonext26ts         |  0  |   nan    |
|        sebotnet33ts_256         |  0  |   nan    |
+---------------------------------+-----+----------+

[yudongsi]

Performance Dashboard for float32 precision -- Single-core Single-thread

Executive Summary

see more

We evaluate different backends across three benchmark suites - torchbench, huggingface and timm. We run these experiments on Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz. Each experiment runs one iteration of forward pass. For accuracy, we check the numerical correctness of forward pass outputs by comparing with native pytorch. We measure speedup by normalizing against the performance of native pytorch. We report mean compilation latency numbers and peak memory footprint reduction ratio.

Caveats

1. Batch size has been reduced to workaround OOM errors. Work is in progress to reduce peak memory footprint.
2. Experiments do not cover dynamic shapes.
3. Experimental setup does not have optimizer.

To measure performance, compilation latency and memory footprint reduction, we remove the models that fail accuracy checks.

Passrate

+----------+------------+-------------+-------------+
| Compiler | torchbench | huggingface | timm_models |
+----------+------------+-------------+-------------+
| inductor | 91%, 49/54 | 100%, 44/44 | 89%, 54/61  |
+----------+------------+-------------+-------------+

Geometric mean speedup

+----------+------------+-------------+-------------+
| Compiler | torchbench | huggingface | timm_models |
+----------+------------+-------------+-------------+
| inductor |    1.04x   |    1.00x    |    1.07x    |
+----------+------------+-------------+-------------+

Mean compilation time (seconds)

+----------+------------+-------------+-------------+
| Compiler | torchbench | huggingface | timm_models |
+----------+------------+-------------+-------------+
| inductor |    17.21   |    19.50    |    22.93    |
+----------+------------+-------------+-------------+

Peak memory footprint compression ratio (higher is better)

+----------+------------+-------------+-------------+
| Compiler | torchbench | huggingface | timm_models |
+----------+------------+-------------+-------------+
| inductor |    1.23x   |     1.30x   |    1.05x    |
+----------+------------+-------------+-------------+

torchbench suite with float32 precision

see more

Performance speedup

+-----------------------------------+----+----------+
|               name                | bs | inductor |
+-----------------------------------+----+----------+
|        shufflenet_v2_x1_0         | 1  |  1.4924  |
|           squeezenet1_1           | 1  |  1.2456  |
|   pytorch_CycleGAN_and_pix2pix    | 1  |  1.1829  |
|             resnet18              | 1  |  1.1615  |
|       functorch_dp_cifar10        | 1  |  1.1532  |
|           pytorch_unet            | 1  |  1.1414  |
| attention_is_all_you_need_pytorch | 1  |  1.1106  |
|            timm_vovnet            | 1  |  1.1089  |
|            Super_SloMo            | 1  |  1.0927  |
|               vgg16               | 1  |  1.0908  |
|              alexnet              | 1  |  1.0802  |
|                drq                | 1  |  1.0768  |
|            timm_regnet            | 1  |  1.0688  |
|        Background_Matting         | 1  |  1.0576  |
|          LearningToPaint          | 1  |  1.0514  |
|               dlrm                | 1  |  1.0468  |
|            densenet121            | 1  |  1.0435  |
|           timm_resnest            | 1  |  1.042   |
|               dcgan               | 1  |  1.0316  |
|          pytorch_stargan          | 16 |  1.0133  |
|              demucs               | 1  |  0.9988  |
|            tts_angular            | 1  |  0.9982  |
|      resnet50_quantized_qat       | 1  |  0.9973  |
|            hf_BigBird             | 1  |  0.9936  |
|    mobilenet_v2_quantized_qat     | 1  |  0.9924  |
|            timm_nfnet             | 1  |  0.9911  |
|      nvidia_deeprecommender       | 1  |  0.9847  |
|           mobilenet_v2            | 1  |  0.9292  |
|            mnasnet1_0             | 1  |  0.9092  |
|              yolov3               | 1  |  0.8915  |
|              hf_GPT2              | 1  |  0.8666  |
|             resnet50              | 1  |  0.8611  |
|           hf_Longformer           | 1  |  0.8127  |
|           BERT_pytorch            | 1  |  0.8125  |
|            hf_T5_large            | 1  |  0.8077  |
|             hf_Albert             | 1  |  0.8057  |
|   timm_vision_transformer_large   | 1  |  0.8052  |
|            hf_Reformer            | 1  |  0.7879  |
|        mobilenet_v3_large         | 1  |  0.7861  |
|           lennard_jones           | 1  |  0.7822  |
|          resnext50_32x4d          | 1  |  0.7543  |
|           hf_GPT2_large           | 1  |  0.7516  |
|               hf_T5               | 1  |  0.7441  |
|           hf_DistilBert           | 1  |  0.7326  |
|              hf_Bart              | 1  |  0.7097  |
|            hf_T5_base             | 1  |  0.6893  |
|              hf_Bert              | 1  |  0.6852  |
|      timm_vision_transformer      | 1  |  0.5967  |
|         timm_efficientnet         | 1  |  0.5114  |
|         soft_actor_critic         | 0  |   0.0    |
|        speech_transformer         | 0  |   0.0    |
|         timm_efficientdet         | 0  |   0.0    |
|           fastNLP_Bert            | 0  |   0.0    |
|             tacotron2             | 0  |   0.0    |
+-----------------------------------+----+----------+

Accuracy

+-----------------------------------+----+------------------+
|               name                | bs |     inductor     |
+-----------------------------------+----+------------------+
|            hf_T5_large            | 1  | pass_due_to_skip |
|           hf_GPT2_large           | 1  | pass_due_to_skip |
|   timm_vision_transformer_large   | 1  | pass_due_to_skip |
|               hf_T5               | 1  |       pass       |
|            hf_Reformer            | 1  |       pass       |
|            hf_T5_base             | 1  |       pass       |
|          LearningToPaint          | 1  |       pass       |
|            Super_SloMo            | 1  |       pass       |
|              alexnet              | 1  |       pass       |
| attention_is_all_you_need_pytorch | 1  |       pass       |
|               dcgan               | 1  |       pass       |
|              demucs               | 1  |       pass       |
|            densenet121            | 1  |       pass       |
|               dlrm                | 1  |       pass       |
|                drq                | 1  |       pass       |
|              yolov3               | 1  |       pass       |
|           mobilenet_v2            | 1  |       pass       |
|             hf_Albert             | 1  |       pass       |
|              hf_Bart              | 1  |       pass       |
|              hf_Bert              | 1  |       pass       |
|            hf_BigBird             | 1  |       pass       |
|           hf_DistilBert           | 1  |       pass       |
|              hf_GPT2              | 1  |       pass       |
|           hf_Longformer           | 1  |       pass       |
|       functorch_dp_cifar10        | 1  |       pass       |
|           lennard_jones           | 1  |       pass       |
|        Background_Matting         | 1  |       pass       |
|          resnext50_32x4d          | 1  |       pass       |
|           BERT_pytorch            | 1  |       pass       |
|      resnet50_quantized_qat       | 1  |       pass       |
|    mobilenet_v2_quantized_qat     | 1  |       pass       |
|        mobilenet_v3_large         | 1  |       pass       |
|      nvidia_deeprecommender       | 1  |       pass       |
|   pytorch_CycleGAN_and_pix2pix    | 1  |       pass       |
|          pytorch_stargan          | 16 |       pass       |
|           pytorch_unet            | 1  |       pass       |
|               vgg16               | 1  |       pass       |
|             resnet50              | 1  |       pass       |
|             resnet18              | 1  |       pass       |
|            mnasnet1_0             | 1  |       pass       |
|        shufflenet_v2_x1_0         | 1  |       pass       |
|           squeezenet1_1           | 1  |       pass       |
|         timm_efficientnet         | 1  |       pass       |
|            timm_nfnet             | 1  |       pass       |
|            timm_regnet            | 1  |       pass       |
|           timm_resnest            | 1  |       pass       |
|      timm_vision_transformer      | 1  |       pass       |
|            timm_vovnet            | 1  |       pass       |
|            tts_angular            | 1  |       pass       |
|         soft_actor_critic         | 0  |      0.0000      |
|           fastNLP_Bert            | 0  |      0.0000      |
|             tacotron2             | 0  |      0.0000      |
|         timm_efficientdet         | 0  |      0.0000      |
|        speech_transformer         | 0  |      0.0000      |
+-----------------------------------+----+------------------+

Compilation latency (sec)

+-----------------------------------+----+----------+
|               name                | bs | inductor |
+-----------------------------------+----+----------+
|            hf_T5_base             | 1  | 76.7783  |
|            hf_T5_large            | 1  | 60.5122  |
|           hf_GPT2_large           | 1  | 59.4694  |
|            densenet121            | 1  | 30.8545  |
|   timm_vision_transformer_large   | 1  | 28.9246  |
|            timm_nfnet             | 1  | 25.4024  |
|              yolov3               | 1  | 24.6705  |
|           hf_Longformer           | 1  | 24.4957  |
|            Super_SloMo            | 1  | 24.3262  |
|         timm_efficientnet         | 1  | 23.1212  |
|        Background_Matting         | 1  | 22.8918  |
|            hf_BigBird             | 1  | 22.3391  |
|           pytorch_unet            | 1  | 18.9934  |
|            timm_regnet            | 1  | 18.4639  |
|            hf_Reformer            | 1  | 18.0788  |
|        mobilenet_v3_large         | 1  | 17.9186  |
|              hf_Bart              | 1  | 17.7819  |
|               hf_T5               | 1  | 17.7063  |
|            timm_vovnet            | 1  | 15.5172  |
|          resnext50_32x4d          | 1  | 14.7342  |
|             resnet50              | 1  | 14.6658  |
|              hf_Bert              | 1  | 14.5967  |
|           mobilenet_v2            | 1  |  14.549  |
|              hf_GPT2              | 1  | 13.9911  |
|           BERT_pytorch            | 1  | 13.8472  |
|           hf_DistilBert           | 1  | 13.8324  |
|           timm_resnest            | 1  |  13.25   |
|            mnasnet1_0             | 1  | 13.1723  |
|        shufflenet_v2_x1_0         | 1  | 12.8739  |
|             hf_Albert             | 1  | 12.6541  |
|      timm_vision_transformer      | 1  | 12.2517  |
| attention_is_all_you_need_pytorch | 1  | 11.9372  |
|       functorch_dp_cifar10        | 1  | 11.8487  |
|          pytorch_stargan          | 16 | 11.5192  |
|          LearningToPaint          | 1  | 10.9144  |
|   pytorch_CycleGAN_and_pix2pix    | 1  | 10.1494  |
|           squeezenet1_1           | 1  |  9.7543  |
|               vgg16               | 1  |  8.5459  |
|             resnet18              | 1  |  8.449   |
|      nvidia_deeprecommender       | 1  |  8.3243  |
|                drq                | 1  |  8.1052  |
|               dlrm                | 1  |  7.9679  |
|              alexnet              | 1  |  7.3684  |
|            tts_angular            | 1  |  6.9832  |
|           lennard_jones           | 1  |  6.5252  |
|              demucs               | 1  |  1.4428  |
|               dcgan               | 1  |  0.2565  |
|    mobilenet_v2_quantized_qat     | 1  |  0.174   |
|      resnet50_quantized_qat       | 1  |  0.1546  |
|           fastNLP_Bert            | 0  |   nan    |
|         soft_actor_critic         | 0  |   nan    |
|        speech_transformer         | 0  |   nan    |
|             tacotron2             | 0  |   nan    |
|         timm_efficientdet         | 0  |   nan    |
+-----------------------------------+----+----------+

Peak Memory Compression Ratio

+-----------------------------------+----+----------+
|               name                | bs | inductor |
+-----------------------------------+----+----------+
|            hf_T5_base             | 1  |  4.3494  |
|           pytorch_unet            | 1  |  2.1317  |
|             hf_Albert             | 1  |  2.1001  |
|           hf_GPT2_large           | 1  |  1.9307  |
|            hf_BigBird             | 1  |  1.7971  |
|            Super_SloMo            | 1  |  1.7855  |
|            hf_T5_large            | 1  |  1.5672  |
|   pytorch_CycleGAN_and_pix2pix    | 1  |  1.4777  |
|               hf_T5               | 1  |  1.3382  |
|              hf_Bart              | 1  |  1.3275  |
|              hf_Bert              | 1  |  1.2964  |
|           hf_Longformer           | 1  |  1.2845  |
|               dlrm                | 1  |  1.2155  |
|          LearningToPaint          | 1  |  1.1866  |
|           hf_DistilBert           | 1  |  1.1789  |
|            timm_nfnet             | 1  |  1.1521  |
|            timm_regnet            | 1  |  1.1463  |
|              hf_GPT2              | 1  |  1.1158  |
|   timm_vision_transformer_large   | 1  |  1.1142  |
|                drq                | 1  |  1.0669  |
|            timm_vovnet            | 1  |  1.0587  |
|           mobilenet_v2            | 1  |  1.0514  |
|         timm_efficientnet         | 1  |  1.0509  |
|      timm_vision_transformer      | 1  |  1.0436  |
|              yolov3               | 1  |  1.0023  |
|              demucs               | 1  |  0.9985  |
|      nvidia_deeprecommender       | 1  |  0.9962  |
|      resnet50_quantized_qat       | 1  |  0.9955  |
|        Background_Matting         | 1  |  0.9953  |
|    mobilenet_v2_quantized_qat     | 1  |  0.9928  |
|            tts_angular            | 1  |  0.9921  |
|          pytorch_stargan          | 16 |  0.9883  |
|            mnasnet1_0             | 1  |  0.9844  |
|            densenet121            | 1  |  0.9831  |
| attention_is_all_you_need_pytorch | 1  |  0.983   |
|           lennard_jones           | 1  |  0.9818  |
|              alexnet              | 1  |  0.9803  |
|        mobilenet_v3_large         | 1  |  0.9771  |
|               vgg16               | 1  |  0.9758  |
|          resnext50_32x4d          | 1  |  0.975   |
|           squeezenet1_1           | 1  |  0.966   |
|               dcgan               | 1  |  0.9639  |
|        shufflenet_v2_x1_0         | 1  |  0.9582  |
|           timm_resnest            | 1  |  0.9582  |
|       functorch_dp_cifar10        | 1  |  0.9542  |
|             resnet50              | 1  |  0.9529  |
|            hf_Reformer            | 1  |  0.9437  |
|           BERT_pytorch            | 1  |  0.9242  |
|             resnet18              | 1  |  0.917   |
|           fastNLP_Bert            | 0  |   nan    |
|         soft_actor_critic         | 0  |   nan    |
|        speech_transformer         | 0  |   nan    |
|             tacotron2             | 0  |   nan    |
|         timm_efficientdet         | 0  |   nan    |
+-----------------------------------+----+----------+

huggingface suite with float32 precision

see more

Performance speedup

+-----------------------------------------+----+----------+
|                  name                   | bs | inductor |
+-----------------------------------------+----+----------+
|     MobileBertForQuestionAnswering      | 1  |  1.0761  |
|            XLNetLMHeadModel             | 1  |  1.049   |
|       AlbertForQuestionAnswering        | 1  |  0.9522  |
|            AlbertForMaskedLM            | 1  |  0.9483  |
|          MobileBertForMaskedLM          | 1  |  0.9464  |
|                 BigBird                 | 1  |  0.9053  |
|     M2M100ForConditionalGeneration      | 1  |  0.9032  |
|             OPTForCausalLM              | 1  |  0.8764  |
|               GoogleFnet                | 1  |  0.8565  |
|            YituTechConvBert             | 1  |  0.8172  |
|         Speech2Text2ForCausalLM         | 1  |  0.8158  |
|    MegatronBertForQuestionAnswering     | 1  |  0.8047  |
|     PegasusForConditionalGeneration     | 1  |  0.8046  |
|       DebertaForQuestionAnswering       | 1  |  0.8035  |
|      MBartForConditionalGeneration      | 1  |  0.8029  |
|         MegatronBertForCausalLM         | 1  |  0.7995  |
|       RobertaForQuestionAnswering       | 1  |  0.7913  |
|          AllenaiLongformerBase          | 1  |  0.791   |
|            TrOCRForCausalLM             | 1  |  0.7903  |
|           PegasusForCausalLM            | 1  |  0.7878  |
|             XGLMForCausalLM             | 1  |  0.7832  |
|            MBartForCausalLM             | 1  |  0.7824  |
|           RobertaForCausalLM            | 1  |  0.7728  |
|     PLBartForConditionalGeneration      | 1  |  0.7699  |
|     DistilBertForQuestionAnswering      | 1  |  0.7654  |
|           DebertaForMaskedLM            | 1  |  0.7648  |
|        BertForQuestionAnswering         | 1  |  0.756   |
|             BertForMaskedLM             | 1  |  0.7524  |
|            PLBartForCausalLM            | 1  |  0.7459  |
|          DistilBertForMaskedLM          | 1  |  0.7431  |
|               DistillGPT2               | 1  |  0.726   |
|       MT5ForConditionalGeneration       | 1  |  0.7145  |
|      GPT2ForSequenceClassification      | 1  |  0.6909  |
|           LayoutLMForMaskedLM           | 1  |  0.6807  |
| BlenderbotSmallForConditionalGeneration | 1  |   0.68   |
|    LayoutLMForSequenceClassification    | 1  |  0.6773  |
|             BartForCausalLM             | 1  |  0.6707  |
|                CamemBert                | 1  |   0.67   |
|       BlenderbotSmallForCausalLM        | 1  |  0.6692  |
|      BartForConditionalGeneration       | 1  |  0.6425  |
|       T5ForConditionalGeneration        | 1  |  0.6322  |
|                 T5Small                 | 1  |  0.6311  |
|       ElectraForQuestionAnswering       | 1  |  0.5028  |
|           ElectraForCausalLM            | 1  |  0.4886  |
+-----------------------------------------+----+----------+

Accuracy

+-----------------------------------------+----+----------+
|                  name                   | bs | inductor |
+-----------------------------------------+----+----------+
|            AlbertForMaskedLM            | 1  |   pass   |
|       AlbertForQuestionAnswering        | 1  |   pass   |
|                CamemBert                | 1  |   pass   |
|          AllenaiLongformerBase          | 1  |   pass   |
|             BartForCausalLM             | 1  |   pass   |
|      BartForConditionalGeneration       | 1  |   pass   |
|             BertForMaskedLM             | 1  |   pass   |
|        BertForQuestionAnswering         | 1  |   pass   |
|                 BigBird                 | 1  |   pass   |
|       BlenderbotSmallForCausalLM        | 1  |   pass   |
| BlenderbotSmallForConditionalGeneration | 1  |   pass   |
|           DebertaForMaskedLM            | 1  |   pass   |
|           LayoutLMForMaskedLM           | 1  |   pass   |
|       DebertaForQuestionAnswering       | 1  |   pass   |
|          DistilBertForMaskedLM          | 1  |   pass   |
|     DistilBertForQuestionAnswering      | 1  |   pass   |
|               DistillGPT2               | 1  |   pass   |
|           ElectraForCausalLM            | 1  |   pass   |
|       ElectraForQuestionAnswering       | 1  |   pass   |
|      GPT2ForSequenceClassification      | 1  |   pass   |
|               GoogleFnet                | 1  |   pass   |
|    LayoutLMForSequenceClassification    | 1  |   pass   |
|     M2M100ForConditionalGeneration      | 1  |   pass   |
|            MBartForCausalLM             | 1  |   pass   |
|     PLBartForConditionalGeneration      | 1  |   pass   |
|      MBartForConditionalGeneration      | 1  |   pass   |
|       MT5ForConditionalGeneration       | 1  |   pass   |
|         MegatronBertForCausalLM         | 1  |   pass   |
|    MegatronBertForQuestionAnswering     | 1  |   pass   |
|          MobileBertForMaskedLM          | 1  |   pass   |
|     MobileBertForQuestionAnswering      | 1  |   pass   |
|             OPTForCausalLM              | 1  |   pass   |
|            PLBartForCausalLM            | 1  |   pass   |
|           PegasusForCausalLM            | 1  |   pass   |
|            XLNetLMHeadModel             | 1  |   pass   |
|     PegasusForConditionalGeneration     | 1  |   pass   |
|           RobertaForCausalLM            | 1  |   pass   |
|       RobertaForQuestionAnswering       | 1  |   pass   |
|         Speech2Text2ForCausalLM         | 1  |   pass   |
|       T5ForConditionalGeneration        | 1  |   pass   |
|                 T5Small                 | 1  |   pass   |
|            TrOCRForCausalLM             | 1  |   pass   |
|             XGLMForCausalLM             | 1  |   pass   |
|            YituTechConvBert             | 1  |   pass   |
+-----------------------------------------+----+----------+

Compilation latency (sec)

+-----------------------------------------+----+----------+
|                  name                   | bs | inductor |
+-----------------------------------------+----+----------+
|      BartForConditionalGeneration       | 1  | 49.6995  |
|            AlbertForMaskedLM            | 1  | 32.9211  |
|          AllenaiLongformerBase          | 1  |  31.457  |
|       AlbertForQuestionAnswering        | 1  | 30.8554  |
|            YituTechConvBert             | 1  | 25.3315  |
|             BartForCausalLM             | 1  | 24.3902  |
|           PegasusForCausalLM            | 1  |  23.914  |
|          MobileBertForMaskedLM          | 1  | 23.6263  |
|            XLNetLMHeadModel             | 1  |  23.513  |
|     PegasusForConditionalGeneration     | 1  | 23.4144  |
|     M2M100ForConditionalGeneration      | 1  | 23.2889  |
|     MobileBertForQuestionAnswering      | 1  | 23.2762  |
|                 BigBird                 | 1  | 22.9858  |
|      MBartForConditionalGeneration      | 1  | 22.3898  |
|           DebertaForMaskedLM            | 1  |  22.357  |
|       T5ForConditionalGeneration        | 1  |  22.172  |
|       MT5ForConditionalGeneration       | 1  | 22.1254  |
|             XGLMForCausalLM             | 1  | 22.0928  |
|                 T5Small                 | 1  | 21.8409  |
|       DebertaForQuestionAnswering       | 1  | 21.3565  |
|         MegatronBertForCausalLM         | 1  | 20.1642  |
|             BertForMaskedLM             | 1  | 20.0772  |
|    MegatronBertForQuestionAnswering     | 1  | 19.6571  |
|      GPT2ForSequenceClassification      | 1  | 19.1087  |
|               GoogleFnet                | 1  | 18.4626  |
| BlenderbotSmallForConditionalGeneration | 1  | 15.2419  |
|                CamemBert                | 1  | 15.1637  |
|     PLBartForConditionalGeneration      | 1  | 14.9998  |
|           LayoutLMForMaskedLM           | 1  | 14.9718  |
|            MBartForCausalLM             | 1  |  14.451  |
|    LayoutLMForSequenceClassification    | 1  | 14.2165  |
|        BertForQuestionAnswering         | 1  | 13.9057  |
|             OPTForCausalLM              | 1  | 13.7452  |
|               DistillGPT2               | 1  |  13.034  |
|            TrOCRForCausalLM             | 1  | 12.9223  |
|           ElectraForCausalLM            | 1  | 12.9222  |
|           RobertaForCausalLM            | 1  | 12.9136  |
|       RobertaForQuestionAnswering       | 1  | 12.6522  |
|       ElectraForQuestionAnswering       | 1  | 12.3606  |
|       BlenderbotSmallForCausalLM        | 1  | 11.2526  |
|         Speech2Text2ForCausalLM         | 1  | 11.1537  |
|            PLBartForCausalLM            | 1  | 10.7109  |
|          DistilBertForMaskedLM          | 1  | 10.5215  |
|     DistilBertForQuestionAnswering      | 1  | 10.2008  |
+-----------------------------------------+----+----------+

Peak Memory Compression Ratio

+-----------------------------------------+----+----------+
|                  name                   | bs | inductor |
+-----------------------------------------+----+----------+
|            AlbertForMaskedLM            | 1  |  2.7784  |
|       AlbertForQuestionAnswering        | 1  |  2.726   |
|      BartForConditionalGeneration       | 1  |  2.1303  |
|       T5ForConditionalGeneration        | 1  |  1.8725  |
|                 T5Small                 | 1  |  1.8719  |
|                 BigBird                 | 1  |  1.8559  |
|      GPT2ForSequenceClassification      | 1  |  1.8462  |
|             BartForCausalLM             | 1  |  1.515   |
|          AllenaiLongformerBase          | 1  |  1.486   |
|            XLNetLMHeadModel             | 1  |  1.3611  |
|       DebertaForQuestionAnswering       | 1  |  1.3577  |
|                CamemBert                | 1  |  1.3398  |
|           LayoutLMForMaskedLM           | 1  |  1.3339  |
|            YituTechConvBert             | 1  |  1.3279  |
|           DebertaForMaskedLM            | 1  |  1.3117  |
|               GoogleFnet                | 1  |  1.2932  |
|    LayoutLMForSequenceClassification    | 1  |  1.2861  |
|           ElectraForCausalLM            | 1  |  1.2708  |
|               DistillGPT2               | 1  |  1.1948  |
|       ElectraForQuestionAnswering       | 1  |  1.1867  |
|     PegasusForConditionalGeneration     | 1  |  1.1209  |
|      MBartForConditionalGeneration      | 1  |  1.0693  |
|       MT5ForConditionalGeneration       | 1  |  1.0645  |
|             BertForMaskedLM             | 1  |  1.0616  |
|     M2M100ForConditionalGeneration      | 1  |  1.0565  |
|         MegatronBertForCausalLM         | 1  |  1.056   |
|           RobertaForCausalLM            | 1  |  1.0498  |
|            TrOCRForCausalLM             | 1  |  1.0498  |
|    MegatronBertForQuestionAnswering     | 1  |  1.0494  |
|       BlenderbotSmallForCausalLM        | 1  |  1.0494  |
|             OPTForCausalLM              | 1  |  1.0467  |
|        BertForQuestionAnswering         | 1  |  1.038   |
|          DistilBertForMaskedLM          | 1  |  1.0359  |
|       RobertaForQuestionAnswering       | 1  |  1.0347  |
|             XGLMForCausalLM             | 1  |  1.0325  |
|     PLBartForConditionalGeneration      | 1  |  1.0286  |
| BlenderbotSmallForConditionalGeneration | 1  |  1.0284  |
|            MBartForCausalLM             | 1  |  1.026   |
|            PLBartForCausalLM            | 1  |  1.0254  |
|     DistilBertForQuestionAnswering      | 1  |  1.017   |
|           PegasusForCausalLM            | 1  |  1.0034  |
|          MobileBertForMaskedLM          | 1  |  0.9964  |
|         Speech2Text2ForCausalLM         | 1  |  0.9816  |
|     MobileBertForQuestionAnswering      | 1  |  0.9741  |
+-----------------------------------------+----+----------+

timm_models suite with float32 precision

see more

Performance speedup

+---------------------------------+----+----------+
|              name               | bs | inductor |
+---------------------------------+----+----------+
|          pnasnet5large          | 1  |  1.5744  |
|           regnety_002           | 1  |  1.3378  |
|        ese_vovnet19b_dw         | 1  |  1.3321  |
|          inception_v3           | 1  |  1.332   |
|       gluon_inception_v3        | 1  |  1.3149  |
|           mnasnet_100           | 1  |  1.3143  |
|        adv_inception_v3         | 1  |  1.3085  |
|            lcnet_050            | 1  |  1.2876  |
|          spnasnet_100           | 1  |  1.2836  |
|           fbnetc_100            | 1  |  1.2526  |
|         mobilenetv2_100         | 1  |  1.2212  |
|            fbnetv3_b            | 1  |   1.16   |
|      mobilenetv3_large_100      | 1  |  1.1506  |
|            gernet_l             | 1  |  1.0846  |
|             dpn107              | 1  |  1.0695  |
|        gluon_xception65         | 1  |  1.0594  |
|          cspdarknet53           | 1  |  1.0591  |
|            repvgg_a2            | 1  |  1.0524  |
|            hrnet_w18            | 1  |  1.0432  |
|          ghostnet_100           | 1  |  1.0058  |
|            nfnet_l0             | 1  |  1.0031  |
|           resnest101e           | 1  |  0.9801  |
|         crossvit_9_240          | 1  |  0.9629  |
|        twins_pcpvt_base         | 1  |  0.9453  |
|           selecsls42b           | 1  |  0.9185  |
|           dm_nfnet_f0           | 1  |  0.9119  |
|         visformer_small         | 1  |  0.9093  |
|        res2net101_26w_4s        | 1  |  0.9033  |
|      xcit_large_24_p8_224       | 1  |  0.9028  |
|          convnext_base          | 1  |  0.8551  |
|        res2net50_14w_8s         | 1  |  0.8483  |
|      beit_base_patch16_224      | 1  |  0.8427  |
|          gmixer_24_224          | 1  |  0.8267  |
| deit_base_distilled_patch16_224 | 1  |   0.79   |
|           res2next50            | 1  |  0.7897  |
|  swin_base_patch4_window7_224   | 1  |  0.7765  |
|           convit_base           | 1  |  0.7703  |
|          cait_m36_384           | 1  |  0.7632  |
|      vit_base_patch16_224       | 1  |  0.761   |
|           volo_d1_224           | 1  |  0.7515  |
|         poolformer_m36          | 1  |  0.736   |
|            mixnet_l             | 1  |  0.729   |
|             dla102              | 1  |  0.7196  |
|           tf_mixnet_l           | 1  |  0.7186  |
|          mixer_b16_224          | 1  |  0.7082  |
|          resmlp_12_224          | 1  |  0.666   |
|            pit_b_224            | 1  |  0.6644  |
|           rexnet_100            | 1  |  0.6578  |
|        tnt_s_patch16_224        | 1  |  0.6416  |
|          gmlp_s16_224           | 1  |  0.6327  |
|          jx_nest_base           | 1  |  0.6187  |
|           mobilevit_s           | 1  |  0.5702  |
|            tinynet_a            | 1  |  0.5466  |
|         coat_lite_mini          | 1  |  0.5378  |
|       tf_efficientnet_b0        | 1  |  0.5215  |
|     swsl_resnext101_32x16d      | 1  |  0.0669  |
|        sebotnet33ts_256         | 0  |   0.0    |
|       eca_botnext26ts_256       | 0  |   0.0    |
|        eca_halonext26ts         | 0  |   0.0    |
|          botnet26t_256          | 0  |   0.0    |
|        convmixer_768_32         | 0  |   0.0    |
+---------------------------------+----+----------+

Accuracy

+---------------------------------+----+---------------+
|              name               | bs |   inductor    |
+---------------------------------+----+---------------+
|        adv_inception_v3         | 1  |     pass      |
|             dpn107              | 1  |     pass      |
|      beit_base_patch16_224      | 1  |     pass      |
|          mixer_b16_224          | 1  |     pass      |
|        ese_vovnet19b_dw         | 1  |     pass      |
|         coat_lite_mini          | 1  |     pass      |
|           convit_base           | 1  |     pass      |
|          convnext_base          | 1  |     pass      |
|         crossvit_9_240          | 1  |     pass      |
|          cspdarknet53           | 1  |     pass      |
| deit_base_distilled_patch16_224 | 1  |     pass      |
|             dla102              | 1  |     pass      |
|           dm_nfnet_f0           | 1  |     pass      |
|            lcnet_050            | 1  |     pass      |
|           volo_d1_224           | 1  |     pass      |
|      xcit_large_24_p8_224       | 1  |     pass      |
|           fbnetc_100            | 1  |     pass      |
|        gluon_xception65         | 1  |     pass      |
|          jx_nest_base           | 1  |     pass      |
|          inception_v3           | 1  |     pass      |
|            hrnet_w18            | 1  |     pass      |
|          gmlp_s16_224           | 1  |     pass      |
|          gmixer_24_224          | 1  |     pass      |
|       gluon_inception_v3        | 1  |     pass      |
|            gernet_l             | 1  |     pass      |
|            fbnetv3_b            | 1  |     pass      |
|           mnasnet_100           | 1  |     pass      |
|            mixnet_l             | 1  |     pass      |
|      vit_base_patch16_224       | 1  |     pass      |
|           res2next50            | 1  |     pass      |
|         mobilenetv2_100         | 1  |     pass      |
|      mobilenetv3_large_100      | 1  |     pass      |
|           mobilevit_s           | 1  |     pass      |
|            nfnet_l0             | 1  |     pass      |
|            pit_b_224            | 1  |     pass      |
|          pnasnet5large          | 1  |     pass      |
|         poolformer_m36          | 1  |     pass      |
|           regnety_002           | 1  |     pass      |
|            repvgg_a2            | 1  |     pass      |
|         visformer_small         | 1  |     pass      |
|        res2net50_14w_8s         | 1  |     pass      |
|        res2net101_26w_4s        | 1  |     pass      |
|          resmlp_12_224          | 1  |     pass      |
|       tf_efficientnet_b0        | 1  |     pass      |
|        twins_pcpvt_base         | 1  |     pass      |
|        tnt_s_patch16_224        | 1  |     pass      |
|           resnest101e           | 1  |     pass      |
|           tf_mixnet_l           | 1  |     pass      |
|            tinynet_a            | 1  |     pass      |
|     swsl_resnext101_32x16d      | 1  |     pass      |
|  swin_base_patch4_window7_224   | 1  |     pass      |
|          spnasnet_100           | 1  |     pass      |
|           selecsls42b           | 1  |     pass      |
|           rexnet_100            | 1  |     pass      |
|        eca_halonext26ts         | 1  |  fail_to_run  |
|        convmixer_768_32         | 1  |  fail_to_run  |
|        sebotnet33ts_256         | 1  |  fail_to_run  |
|          botnet26t_256          | 1  |  fail_to_run  |
|       eca_botnext26ts_256       | 1  |  fail_to_run  |
|          ghostnet_100           | 1  | fail_accuracy |
|          cait_m36_384           | 1  | fail_accuracy |
+---------------------------------+----+---------------+

Compilation latency (sec)

+---------------------------------+----+----------+
|              name               | bs | inductor |
+---------------------------------+----+----------+
|     swsl_resnext101_32x16d      | 1  | 85.6242  |
|          pnasnet5large          | 1  | 43.6874  |
|            fbnetv3_b            | 1  | 34.4074  |
|          cait_m36_384           | 1  | 33.7081  |
|           tf_mixnet_l           | 1  | 33.6983  |
|        twins_pcpvt_base         | 1  | 31.5227  |
|            hrnet_w18            | 1  | 31.1337  |
|           rexnet_100            | 1  | 30.7918  |
|  swin_base_patch4_window7_224   | 1  | 30.3129  |
|           mobilevit_s           | 1  | 30.2527  |
|            mixnet_l             | 1  |  29.901  |
|      xcit_large_24_p8_224       | 1  | 28.4048  |
|             dpn107              | 1  | 26.7877  |
|        adv_inception_v3         | 1  | 26.4651  |
|        res2net50_14w_8s         | 1  | 26.3563  |
|           dm_nfnet_f0           | 1  | 26.1631  |
|          ghostnet_100           | 1  | 25.6518  |
|         poolformer_m36          | 1  | 24.9019  |
|       tf_efficientnet_b0        | 1  | 24.8772  |
|           resnest101e           | 1  | 24.5232  |
|         visformer_small         | 1  | 24.3597  |
|            tinynet_a            | 1  | 24.2595  |
|        res2net101_26w_4s        | 1  | 23.5626  |
|          jx_nest_base           | 1  | 23.1414  |
|         coat_lite_mini          | 1  | 22.4646  |
|            nfnet_l0             | 1  | 22.4105  |
|      mobilenetv3_large_100      | 1  | 22.3182  |
|             dla102              | 1  | 21.2783  |
|           volo_d1_224           | 1  | 21.2219  |
|         crossvit_9_240          | 1  | 21.2091  |
|        tnt_s_patch16_224        | 1  | 21.0931  |
|          convnext_base          | 1  | 19.4932  |
|          cspdarknet53           | 1  | 19.3065  |
|           fbnetc_100            | 1  | 18.9707  |
|       gluon_inception_v3        | 1  | 18.6493  |
|           res2next50            | 1  | 18.5211  |
|            pit_b_224            | 1  | 18.1465  |
|          spnasnet_100           | 1  | 18.1066  |
|          inception_v3           | 1  | 17.9952  |
|          gmlp_s16_224           | 1  | 17.9558  |
|      beit_base_patch16_224      | 1  |  17.513  |
|           regnety_002           | 1  | 17.0633  |
|        gluon_xception65         | 1  | 16.8521  |
|           mnasnet_100           | 1  | 16.5367  |
|         mobilenetv2_100         | 1  | 16.3189  |
|           convit_base           | 1  | 16.2035  |
|            gernet_l             | 1  | 14.7975  |
|          gmixer_24_224          | 1  |  14.685  |
|           selecsls42b           | 1  | 14.2609  |
|        ese_vovnet19b_dw         | 1  | 13.4973  |
|          mixer_b16_224          | 1  |  13.462  |
|      vit_base_patch16_224       | 1  | 13.3318  |
| deit_base_distilled_patch16_224 | 1  | 13.0111  |
|            repvgg_a2            | 1  | 12.9665  |
|            lcnet_050            | 1  | 12.8088  |
|          resmlp_12_224          | 1  | 10.4233  |
|          botnet26t_256          | 0  |   nan    |
|        convmixer_768_32         | 0  |   nan    |
|       eca_botnext26ts_256       | 0  |   nan    |
|        eca_halonext26ts         | 0  |   nan    |
|        sebotnet33ts_256         | 0  |   nan    |
+---------------------------------+----+----------+

Peak Memory Compression Ratio

+---------------------------------+----+----------+
|              name               | bs | inductor |
+---------------------------------+----+----------+
|      xcit_large_24_p8_224       | 1  |  2.2814  |
|          cait_m36_384           | 1  |  2.1693  |
|           dm_nfnet_f0           | 1  |  1.2079  |
|           mobilevit_s           | 1  |  1.2047  |
|          jx_nest_base           | 1  |  1.1988  |
|            nfnet_l0             | 1  |  1.1854  |
|             dpn107              | 1  |  1.1766  |
|        gluon_xception65         | 1  |  1.1551  |
|          convnext_base          | 1  |  1.155   |
|           convit_base           | 1  |  1.1399  |
|          cspdarknet53           | 1  |  1.1261  |
|         poolformer_m36          | 1  |  1.1142  |
|  swin_base_patch4_window7_224   | 1  |  1.106   |
|            pit_b_224            | 1  |  1.1013  |
|      beit_base_patch16_224      | 1  |  1.0972  |
|        twins_pcpvt_base         | 1  |  1.0907  |
|        tnt_s_patch16_224        | 1  |  1.0873  |
|           volo_d1_224           | 1  |  1.087   |
| deit_base_distilled_patch16_224 | 1  |  1.082   |
|      vit_base_patch16_224       | 1  |  1.0789  |
|          mixer_b16_224          | 1  |  1.0786  |
|         mobilenetv2_100         | 1  |  1.0611  |
|        ese_vovnet19b_dw         | 1  |  1.0596  |
|       tf_efficientnet_b0        | 1  |  1.057   |
|          resmlp_12_224          | 1  |  1.0457  |
|         coat_lite_mini          | 1  |  1.0422  |
|           rexnet_100            | 1  |  1.0408  |
|            mixnet_l             | 1  |  1.0377  |
|            gernet_l             | 1  |  1.0281  |
|            fbnetv3_b            | 1  |  1.0236  |
|            tinynet_a            | 1  |  1.0189  |
|           fbnetc_100            | 1  |  1.0121  |
|          spnasnet_100           | 1  |  1.0098  |
|           mnasnet_100           | 1  |  1.0097  |
|            repvgg_a2            | 1  |  1.0028  |
|         crossvit_9_240          | 1  |  0.9982  |
|           resnest101e           | 1  |  0.9928  |
|      mobilenetv3_large_100      | 1  |  0.9677  |
|            lcnet_050            | 1  |  0.9643  |
|           regnety_002           | 1  |  0.9604  |
|         visformer_small         | 1  |  0.9598  |
|          inception_v3           | 1  |  0.9554  |
|           res2next50            | 1  |  0.9538  |
|             dla102              | 1  |  0.9508  |
|        adv_inception_v3         | 1  |  0.9508  |
|            hrnet_w18            | 1  |  0.9504  |
|          ghostnet_100           | 1  |  0.9456  |
|        res2net50_14w_8s         | 1  |  0.9426  |
|       gluon_inception_v3        | 1  |  0.9393  |
|          gmixer_24_224          | 1  |  0.9237  |
|        res2net101_26w_4s        | 1  |  0.9135  |
|          gmlp_s16_224           | 1  |  0.8851  |
|     swsl_resnext101_32x16d      | 1  |  0.8458  |
|           tf_mixnet_l           | 1  |  0.8379  |
|           selecsls42b           | 1  |  0.8177  |
|          pnasnet5large          | 1  |  0.7496  |
|          botnet26t_256          | 0  |   nan    |
|        convmixer_768_32         | 0  |   nan    |
|       eca_botnext26ts_256       | 0  |   nan    |
|        eca_halonext26ts         | 0  |   nan    |
|        sebotnet33ts_256         | 0  |   nan    |
+---------------------------------+----+----------+

[chuanqi129]

Performance Dashboard for float32 precision -- Single-Socket Multi-threads (2022-11-09 nightly release)

Executive Summary

see more

We evaluate different backends across three benchmark suites - torchbench, huggingface and timm. We run these experiments on Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz. Each experiment runs one iteration of forward pass. For accuracy, we check the numerical correctness of forward pass outputs by comparing with native pytorch. We measure speedup by normalizing against the performance of native pytorch. We report mean compilation latency numbers and peak memory footprint reduction ratio.

Caveats

1. Batch size has been reduced to workaround OOM errors. Work is in progress to reduce peak memory footprint.
2. Experiments do not cover dynamic shapes.
3. Experimental setup does not have optimizer.

SW information

SW	Nightly commit	Master/Main commit
Pytorch	3b29687	a7420d2
Torchbench	/	022dfe3
torchaudio	4b10b6a	74f9a89
torchtext	71e4561	c047efe
torchvision	797e1ac	ffd5a56

HW information

Item	Value
Manufacturer	Amazon EC2
Product Name	c6i.16xlarge
CPU Model	Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz
Installed Memory	128GB (1x128GB DDR4 3200 MT/s [Unknown])
OS	Ubuntu 20.04.5 LTS
Kernel	5.15.0-1022-aws
Microcode	0xd000331
GCC	gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
GLIBC	ldd (Ubuntu GLIBC 2.31-0ubuntu9.9) 2.31
Binutils	GNU ld (GNU Binutils for Ubuntu) 2.34
Python	Python 3.8.13
OpenSSL	OpenSSL 1.1.1s 1 Nov 2022

To measure performance, compilation latency and memory footprint reduction, we remove the models that fail accuracy checks.

Passrate

+----------+-------------+-------------+-------------+
| Compiler | torchbench  | huggingface | timm_models |
+----------+-------------+-------------+-------------+
| inductor | 100%, 55/55 | 100%, 44/44 | 92%, 56/61  |
+----------+-------------+-------------+-------------+

Geometric mean speedup

+----------+------------+-------------+-------------+
| Compiler | torchbench | huggingface | timm_models |
+----------+------------+-------------+-------------+
| inductor |   1.08x    |    1.07x    |    1.09x    |
+----------+------------+-------------+-------------+

Mean compilation time (seconds)

+----------+------------+-------------+-------------+
| Compiler | torchbench | huggingface | timm_models |
+----------+------------+-------------+-------------+
| inductor |   14.89    |    16.47    |    20.62    |
+----------+------------+-------------+-------------+

Peak memory footprint compression ratio (higher is better)

+----------+------------+-------------+-------------+
| Compiler | torchbench | huggingface | timm_models |
+----------+------------+-------------+-------------+
| inductor |   0.95x    |    0.97x    |    0.99x    |
+----------+------------+-------------+-------------+

torchbench suite with float32 precision

see more

Performance speedup

+-----------------------------------+------+----------+
|               name                |  bs  | inductor |
+-----------------------------------+------+----------+
|        shufflenet_v2_x1_0         |  64  |  1.8287  |
|         soft_actor_critic         | 256  |  1.6029  |
|            densenet121            |  64  |   1.46   |
|            mnasnet1_0             |  32  |  1.2959  |
|             resnet18              |  8   |  1.2223  |
|           pytorch_unet            |  1   |  1.1886  |
|       functorch_dp_cifar10        |  64  |  1.1859  |
|              alexnet              | 128  |  1.1594  |
|           squeezenet1_1           |  16  |  1.1573  |
|             resnet152             |  32  |  1.1529  |
|           mobilenet_v2            |  16  |  1.1485  |
|           timm_resnest            |  32  |  1.1455  |
|        mobilenet_v3_large         |  32  |  1.1413  |
|   pytorch_CycleGAN_and_pix2pix    |  1   |  1.1121  |
|            timm_vovnet            |  32  |  1.1105  |
|             resnet50              |  32  |  1.0939  |
|               dcgan               | 256  |  1.0811  |
|               dlrm                | 2048 |  1.081   |
|            Super_SloMo            |  6   |  1.0735  |
|          LearningToPaint          |  96  |  1.0641  |
|        Background_Matting         |  1   |  1.0627  |
|               vgg16               |  4   |  1.0589  |
|           BERT_pytorch            |  2   |  1.0586  |
|            timm_regnet            |  32  |  1.0365  |
|            hf_T5_large            |  1   |  1.036   |
|          pytorch_stargan          |  16  |  1.0281  |
|     detectron2_fcos_r_50_fpn      |  1   |  1.0179  |
|            hf_Reformer            |  1   |  1.0089  |
|            tts_angular            |  64  |  1.0026  |
|            hf_BigBird             |  1   |  0.9993  |
|              demucs               |  1   |  0.999   |
|    mobilenet_v2_quantized_qat     |  96  |  0.9988  |
|                drq                |  1   |  0.9979  |
|      resnet50_quantized_qat       |  32  |  0.9952  |
|           hf_Longformer           |  1   |  0.9891  |
|               hf_T5               |  1   |  0.978   |
|          vision_maskrcnn          |  1   |  0.9622  |
|        speech_transformer         |  1   |  0.9319  |
|           hf_GPT2_large           |  1   |  0.9271  |
|              hf_GPT2              |  1   |  0.9202  |
|   timm_vision_transformer_large   |  8   |  0.9158  |
|            timm_nfnet             | 128  |  0.9154  |
|           hf_DistilBert           |  1   |  0.8986  |
|             hf_Albert             |  1   |  0.8814  |
|              hf_Bert              |  1   |  0.8779  |
|              yolov3               |  8   |  0.8727  |
|           fastNLP_Bert            |  1   |  0.8721  |
|              hf_Bart              |  1   |  0.8498  |
|          resnext50_32x4d          |  8   |  0.8362  |
|      nvidia_deeprecommender       | 256  |  0.8304  |
|            hf_T5_base             |  1   |  0.8286  |
|      timm_vision_transformer      |  8   |  0.764   |
|         timm_efficientnet         |  64  |  0.7439  |
| attention_is_all_you_need_pytorch |  32  |  0.7132  |
|           lennard_jones           | 1000 |  0.3489  |
+-----------------------------------+------+----------+

Accuracy

+-----------------------------------+-----+------------------+
|               name                | bs  |     inductor     |
+-----------------------------------+-----+------------------+
|            hf_T5_large            |  2  | pass_due_to_skip |
|           hf_GPT2_large           |  2  | pass_due_to_skip |
|   timm_vision_transformer_large   |  2  | pass_due_to_skip |
|              yolov3               |  2  |       pass       |
|                drq                |  1  |       pass       |
|        Background_Matting         |  1  |       pass       |
|       functorch_dp_cifar10        |  2  |       pass       |
|          LearningToPaint          |  2  |       pass       |
|            Super_SloMo            |  2  |       pass       |
|              alexnet              |  2  |       pass       |
| attention_is_all_you_need_pytorch |  2  |       pass       |
|               dcgan               |  2  |       pass       |
|              demucs               |  1  |       pass       |
|            densenet121            |  2  |       pass       |
|               dlrm                |  2  |       pass       |
|           lennard_jones           |  2  |       pass       |
|            hf_T5_base             |  2  |       pass       |
|            mnasnet1_0             |  2  |       pass       |
|             hf_Albert             |  2  |       pass       |
|              hf_Bart              |  2  |       pass       |
|              hf_Bert              |  2  |       pass       |
|            hf_BigBird             |  2  |       pass       |
|           hf_DistilBert           |  2  |       pass       |
|              hf_GPT2              |  2  |       pass       |
|           hf_Longformer           |  2  |       pass       |
|            hf_Reformer            |  2  |       pass       |
|               hf_T5               |  2  |       pass       |
|           fastNLP_Bert            |  2  |       pass       |
|    mobilenet_v2_quantized_qat     |  2  |       pass       |
|               vgg16               |  2  |       pass       |
|            tts_angular            |  2  |       pass       |
|           BERT_pytorch            |  2  |       pass       |
|          resnext50_32x4d          |  2  |       pass       |
|        mobilenet_v3_large         |  2  |       pass       |
|      nvidia_deeprecommender       |  2  |       pass       |
|   pytorch_CycleGAN_and_pix2pix    |  1  |       pass       |
|          pytorch_stargan          | 16  |       pass       |
|           pytorch_unet            |  2  |       pass       |
|             resnet152             |  2  |       pass       |
|             resnet18              |  2  |       pass       |
|             resnet50              |  2  |       pass       |
|      resnet50_quantized_qat       |  2  |       pass       |
|        shufflenet_v2_x1_0         |  2  |       pass       |
|           mobilenet_v2            |  2  |       pass       |
|         soft_actor_critic         | 256 |       pass       |
|        speech_transformer         |  2  |       pass       |
|           squeezenet1_1           |  2  |       pass       |
|         timm_efficientnet         |  2  |       pass       |
|            timm_nfnet             |  2  |       pass       |
|            timm_regnet            |  2  |       pass       |
|           timm_resnest            |  2  |       pass       |
|      timm_vision_transformer      |  2  |       pass       |
|            timm_vovnet            |  2  |       pass       |
|     detectron2_fcos_r_50_fpn      |  2  |   fail_to_run    |
|          vision_maskrcnn          |  0  |      0.0000      |
+-----------------------------------+-----+------------------+

Compilation latency (sec)

+-----------------------------------+------+----------+
|               name                |  bs  | inductor |
+-----------------------------------+------+----------+
|     detectron2_fcos_r_50_fpn      |  1   | 64.5524  |
|          vision_maskrcnn          |  1   | 63.9086  |
|           BERT_pytorch            |  2   | 45.7082  |
|            hf_T5_large            |  1   | 33.6462  |
|           hf_GPT2_large           |  1   | 22.5423  |
|            hf_BigBird             |  1   | 22.4972  |
|            hf_T5_base             |  1   | 22.2742  |
|            timm_nfnet             | 128  | 22.1908  |
|   timm_vision_transformer_large   |  8   | 21.7555  |
|           hf_Longformer           |  1   | 21.6172  |
|            densenet121            |  64  | 21.4901  |
|              yolov3               |  8   |  19.69   |
|           mobilenet_v2            |  16  | 18.9421  |
|            Super_SloMo            |  6   | 17.2204  |
|             resnet152             |  32  | 17.0653  |
|        speech_transformer         |  1   | 15.5597  |
|            timm_regnet            |  32  |  15.255  |
|              hf_Bart              |  1   | 13.7245  |
|         timm_efficientnet         |  64  | 13.6015  |
|            hf_Reformer            |  1   | 13.5318  |
|           fastNLP_Bert            |  1   | 13.1859  |
| attention_is_all_you_need_pytorch |  32  | 13.0332  |
|               hf_T5               |  1   | 12.8551  |
|              hf_Bert              |  1   | 12.2325  |
|              hf_GPT2              |  1   | 11.7985  |
|      timm_vision_transformer      |  8   | 11.7079  |
|        Background_Matting         |  1   | 11.3904  |
|             hf_Albert             |  1   | 11.3643  |
|            timm_vovnet            |  32  | 11.1937  |
|             resnet50              |  32  | 11.0994  |
|        mobilenet_v3_large         |  32  | 10.9908  |
|          resnext50_32x4d          |  8   | 10.7624  |
|        shufflenet_v2_x1_0         |  64  | 10.6591  |
|            mnasnet1_0             |  32  | 10.2952  |
|           hf_DistilBert           |  1   | 10.1395  |
|           timm_resnest            |  32  |  9.9293  |
|       functorch_dp_cifar10        |  64  |  9.5011  |
|           pytorch_unet            |  1   |  9.3573  |
|          pytorch_stargan          |  16  |  9.3224  |
|   pytorch_CycleGAN_and_pix2pix    |  1   |  9.2365  |
|          LearningToPaint          |  96  |  8.6886  |
|           squeezenet1_1           |  16  |  8.6509  |
|             resnet18              |  8   |  8.522   |
|      nvidia_deeprecommender       | 256  |  8.3779  |
|               vgg16               |  4   |  8.2719  |
|               dlrm                | 2048 |  8.2115  |
|              alexnet              | 128  |  8.1818  |
|            tts_angular            |  64  |  8.0338  |
|                drq                |  1   |  7.9702  |
|         soft_actor_critic         | 256  |  7.7336  |
|           lennard_jones           | 1000 |  7.7074  |
|              demucs               |  1   |  1.2856  |
|               dcgan               | 256  |  0.3822  |
|    mobilenet_v2_quantized_qat     |  96  |  0.0985  |
|      resnet50_quantized_qat       |  32  |  0.0088  |
+-----------------------------------+------+----------+

Peak Memory Compression Ratio

+-----------------------------------+------+----------+
|               name                |  bs  | inductor |
+-----------------------------------+------+----------+
|              demucs               |  1   |  0.9988  |
|               dlrm                | 2048 |  0.9971  |
|      resnet50_quantized_qat       |  32  |  0.996   |
|        Background_Matting         |  1   |  0.9938  |
|            timm_regnet            |  32  |  0.9936  |
|               vgg16               |  4   |  0.9927  |
|            timm_nfnet             | 128  |  0.9917  |
|          LearningToPaint          |  96  |  0.9912  |
|            hf_BigBird             |  1   |  0.9895  |
|            densenet121            |  64  |  0.9893  |
|             resnet152             |  32  |  0.9889  |
|             resnet50              |  32  |  0.9866  |
|           pytorch_unet            |  1   |  0.986   |
|   timm_vision_transformer_large   |  8   |  0.9859  |
|            mnasnet1_0             |  32  |  0.9852  |
|           lennard_jones           | 1000 |  0.9837  |
|           mobilenet_v2            |  16  |  0.9828  |
|         soft_actor_critic         | 256  |  0.9828  |
|                drq                |  1   |  0.9819  |
| attention_is_all_you_need_pytorch |  32  |  0.9818  |
|        mobilenet_v3_large         |  32  |  0.9814  |
|        shufflenet_v2_x1_0         |  64  |  0.9813  |
|           hf_GPT2_large           |  1   |  0.9793  |
|            Super_SloMo            |  6   |  0.9789  |
|              hf_Bart              |  1   |  0.9772  |
|     detectron2_fcos_r_50_fpn      |  1   |  0.9771  |
|            timm_vovnet            |  32  |  0.9767  |
|           hf_DistilBert           |  1   |  0.9761  |
|    mobilenet_v2_quantized_qat     |  96  |  0.9748  |
|             hf_Albert             |  1   |  0.9741  |
|           squeezenet1_1           |  16  |  0.9733  |
|          resnext50_32x4d          |  8   |  0.9732  |
|            tts_angular            |  64  |  0.9722  |
|           BERT_pytorch            |  2   |  0.9715  |
|         timm_efficientnet         |  64  |  0.9707  |
|        speech_transformer         |  1   |  0.9686  |
|          vision_maskrcnn          |  1   |  0.9681  |
|              hf_GPT2              |  1   |  0.9645  |
|          pytorch_stargan          |  16  |  0.9623  |
|               dcgan               | 256  |  0.957   |
|             resnet18              |  8   |  0.9524  |
|   pytorch_CycleGAN_and_pix2pix    |  1   |  0.951   |
|            hf_T5_base             |  1   |  0.9448  |
|      timm_vision_transformer      |  8   |  0.9343  |
|            hf_Reformer            |  1   |  0.9338  |
|              alexnet              | 128  |  0.9295  |
|           fastNLP_Bert            |  1   |  0.9217  |
|              yolov3               |  8   |  0.9142  |
|       functorch_dp_cifar10        |  64  |  0.9129  |
|           timm_resnest            |  32  |  0.8811  |
|           hf_Longformer           |  1   |  0.877   |
|              hf_Bert              |  1   |  0.7856  |
|            hf_T5_large            |  1   |  0.6783  |
|               hf_T5               |  1   |  0.6081  |
|      nvidia_deeprecommender       | 256  |  0.5844  |
+-----------------------------------+------+----------+

huggingface suite with float32 precision

see more

Performance speedup

+-----------------------------------------+-----+----------+
|                  name                   | bs  | inductor |
+-----------------------------------------+-----+----------+
|       MT5ForConditionalGeneration       |  8  |  2.4476  |
|            XLNetLMHeadModel             | 32  |  1.9966  |
|            YituTechConvBert             |  1  |  1.3041  |
|               DistillGPT2               |  1  |  1.2765  |
|     MobileBertForQuestionAnswering      | 64  |  1.2614  |
|               GoogleFnet                |  1  |  1.2128  |
|          MobileBertForMaskedLM          | 32  |  1.0902  |
|                 BigBird                 |  1  |  1.0684  |
|             XGLMForCausalLM             |  8  |  1.0601  |
|     M2M100ForConditionalGeneration      |  8  |  1.0597  |
|          AllenaiLongformerBase          |  1  |  1.0388  |
|                CamemBert                |  1  |  1.0358  |
|            AlbertForMaskedLM            |  4  |  1.0232  |
|       AlbertForQuestionAnswering        |  4  |  1.0218  |
|             OPTForCausalLM              | 32  |  0.9935  |
|           DebertaForMaskedLM            |  4  |  0.9902  |
|                 T5Small                 |  1  |  0.9826  |
|       DebertaForQuestionAnswering       |  8  |  0.9465  |
|     PLBartForConditionalGeneration      | 16  |  0.9233  |
|      MBartForConditionalGeneration      | 16  |  0.9222  |
|         MegatronBertForCausalLM         | 16  |  0.9217  |
|     PegasusForConditionalGeneration     | 16  |  0.9195  |
|      GPT2ForSequenceClassification      |  4  |  0.9192  |
|    MegatronBertForQuestionAnswering     | 16  |  0.9112  |
|    LayoutLMForSequenceClassification    | 16  |  0.9082  |
|           RobertaForCausalLM            | 64  |  0.9069  |
|         Speech2Text2ForCausalLM         | 128 |  0.9055  |
|            TrOCRForCausalLM             | 32  |  0.9011  |
|           ElectraForCausalLM            | 32  |  0.8992  |
|     DistilBertForQuestionAnswering      | 64  |  0.8984  |
|            PLBartForCausalLM            | 32  |  0.8943  |
|           PegasusForCausalLM            | 32  |  0.8924  |
|       RobertaForQuestionAnswering       | 128 |  0.8892  |
|           LayoutLMForMaskedLM           | 16  |  0.8876  |
|             BertForMaskedLM             | 64  |  0.8794  |
|        BertForQuestionAnswering         | 128 |  0.8784  |
|       ElectraForQuestionAnswering       | 64  |  0.8612  |
|          DistilBertForMaskedLM          | 64  |  0.8588  |
|       T5ForConditionalGeneration        |  4  |  0.8556  |
| BlenderbotSmallForConditionalGeneration | 64  |  0.8261  |
|             BartForCausalLM             |  4  |  0.8217  |
|      BartForConditionalGeneration       |  2  |  0.8188  |
|       BlenderbotSmallForCausalLM        | 64  |  0.8048  |
|            MBartForCausalLM             | 32  |  0.7923  |
+-----------------------------------------+-----+----------+

Accuracy

+-----------------------------------------+----+----------+
|                  name                   | bs | inductor |
+-----------------------------------------+----+----------+
|            AlbertForMaskedLM            | 1  |   pass   |
|       AlbertForQuestionAnswering        | 1  |   pass   |
|                CamemBert                | 1  |   pass   |
|          AllenaiLongformerBase          | 1  |   pass   |
|             BartForCausalLM             | 1  |   pass   |
|      BartForConditionalGeneration       | 1  |   pass   |
|             BertForMaskedLM             | 1  |   pass   |
|        BertForQuestionAnswering         | 1  |   pass   |
|                 BigBird                 | 1  |   pass   |
|       BlenderbotSmallForCausalLM        | 1  |   pass   |
| BlenderbotSmallForConditionalGeneration | 1  |   pass   |
|           DebertaForMaskedLM            | 1  |   pass   |
|           LayoutLMForMaskedLM           | 1  |   pass   |
|       DebertaForQuestionAnswering       | 1  |   pass   |
|          DistilBertForMaskedLM          | 1  |   pass   |
|     DistilBertForQuestionAnswering      | 1  |   pass   |
|               DistillGPT2               | 1  |   pass   |
|           ElectraForCausalLM            | 1  |   pass   |
|       ElectraForQuestionAnswering       | 1  |   pass   |
|      GPT2ForSequenceClassification      | 1  |   pass   |
|               GoogleFnet                | 1  |   pass   |
|    LayoutLMForSequenceClassification    | 1  |   pass   |
|     M2M100ForConditionalGeneration      | 1  |   pass   |
|            MBartForCausalLM             | 1  |   pass   |
|     PLBartForConditionalGeneration      | 1  |   pass   |
|      MBartForConditionalGeneration      | 1  |   pass   |
|       MT5ForConditionalGeneration       | 1  |   pass   |
|         MegatronBertForCausalLM         | 1  |   pass   |
|    MegatronBertForQuestionAnswering     | 1  |   pass   |
|          MobileBertForMaskedLM          | 1  |   pass   |
|     MobileBertForQuestionAnswering      | 1  |   pass   |
|             OPTForCausalLM              | 1  |   pass   |
|            PLBartForCausalLM            | 1  |   pass   |
|           PegasusForCausalLM            | 1  |   pass   |
|            XLNetLMHeadModel             | 1  |   pass   |
|     PegasusForConditionalGeneration     | 1  |   pass   |
|           RobertaForCausalLM            | 1  |   pass   |
|       RobertaForQuestionAnswering       | 1  |   pass   |
|         Speech2Text2ForCausalLM         | 1  |   pass   |
|       T5ForConditionalGeneration        | 1  |   pass   |
|                 T5Small                 | 1  |   pass   |
|            TrOCRForCausalLM             | 1  |   pass   |
|             XGLMForCausalLM             | 1  |   pass   |
|            YituTechConvBert             | 1  |   pass   |
+-----------------------------------------+----+----------+

Compilation latency (sec)

+-----------------------------------------+-----+----------+
|                  name                   | bs  | inductor |
+-----------------------------------------+-----+----------+
|     PegasusForConditionalGeneration     | 16  | 23.9196  |
|          MobileBertForMaskedLM          | 32  | 23.8928  |
|          AllenaiLongformerBase          |  1  | 23.1661  |
|     MobileBertForQuestionAnswering      | 64  | 22.9028  |
|           DebertaForMaskedLM            |  4  | 22.8131  |
|       DebertaForQuestionAnswering       |  8  | 22.6847  |
|     M2M100ForConditionalGeneration      |  8  | 22.5781  |
|      BartForConditionalGeneration       |  2  | 21.6736  |
|                 BigBird                 |  1  | 21.5731  |
|      MBartForConditionalGeneration      | 16  | 21.0786  |
|             XGLMForCausalLM             |  8  | 20.0701  |
|          DistilBertForMaskedLM          | 64  |  19.039  |
|         MegatronBertForCausalLM         | 16  | 18.8882  |
|       AlbertForQuestionAnswering        |  4  | 18.6163  |
|    MegatronBertForQuestionAnswering     | 16  | 18.5201  |
|            YituTechConvBert             |  1  | 17.9706  |
| BlenderbotSmallForConditionalGeneration | 64  | 17.3304  |
|           PegasusForCausalLM            | 32  | 16.4097  |
|            AlbertForMaskedLM            |  4  | 16.1552  |
|       MT5ForConditionalGeneration       |  8  | 15.8319  |
|       RobertaForQuestionAnswering       | 128 | 15.5256  |
|        BertForQuestionAnswering         | 128 | 15.4197  |
|           LayoutLMForMaskedLM           | 16  | 15.2302  |
|       T5ForConditionalGeneration        |  4  | 14.9184  |
|           RobertaForCausalLM            | 64  | 14.8398  |
|             BartForCausalLM             |  4  |  14.813  |
|       ElectraForQuestionAnswering       | 64  | 14.6766  |
|             BertForMaskedLM             | 64  |  14.621  |
|    LayoutLMForSequenceClassification    | 16  | 14.5713  |
|           ElectraForCausalLM            | 32  | 14.4872  |
|     PLBartForConditionalGeneration      | 16  | 14.4639  |
|            MBartForCausalLM             | 32  | 14.2826  |
|      GPT2ForSequenceClassification      |  4  | 14.1216  |
|            TrOCRForCausalLM             | 32  | 13.7212  |
|             OPTForCausalLM              | 32  | 13.2485  |
|                 T5Small                 |  1  |  13.094  |
|                CamemBert                |  1  | 12.6413  |
|               GoogleFnet                |  1  | 12.4936  |
|         Speech2Text2ForCausalLM         | 128 | 12.3975  |
|       BlenderbotSmallForCausalLM        | 64  | 12.2115  |
|     DistilBertForQuestionAnswering      | 64  | 11.0523  |
|            PLBartForCausalLM            | 32  |  10.871  |
|               DistillGPT2               |  1  |  9.8446  |
|            XLNetLMHeadModel             | 32  |  5.9197  |
+-----------------------------------------+-----+----------+

Peak Memory Compression Ratio

+-----------------------------------------+-----+----------+
|                  name                   | bs  | inductor |
+-----------------------------------------+-----+----------+
|            AlbertForMaskedLM            |  4  |  0.9977  |
|       AlbertForQuestionAnswering        |  4  |  0.9976  |
|           ElectraForCausalLM            | 32  |  0.9961  |
|             BartForCausalLM             |  4  |  0.9959  |
|           RobertaForCausalLM            | 64  |  0.9955  |
|       BlenderbotSmallForCausalLM        | 64  |  0.9954  |
|             BertForMaskedLM             | 64  |  0.9952  |
|       ElectraForQuestionAnswering       | 64  |  0.995   |
| BlenderbotSmallForConditionalGeneration | 64  |  0.9948  |
|      GPT2ForSequenceClassification      |  4  |  0.9945  |
|          DistilBertForMaskedLM          | 64  |  0.9944  |
|            TrOCRForCausalLM             | 32  |  0.9941  |
|           PegasusForCausalLM            | 32  |  0.9939  |
|       T5ForConditionalGeneration        |  4  |  0.9936  |
|            MBartForCausalLM             | 32  |  0.9932  |
|             OPTForCausalLM              | 32  |  0.9931  |
|            PLBartForCausalLM            | 32  |  0.9925  |
|         Speech2Text2ForCausalLM         | 128 |  0.9925  |
|      BartForConditionalGeneration       |  2  |  0.9919  |
|     DistilBertForQuestionAnswering      | 64  |  0.9915  |
|           DebertaForMaskedLM            |  4  |  0.9915  |
|     PegasusForConditionalGeneration     | 16  |  0.9914  |
|    LayoutLMForSequenceClassification    | 16  |  0.9913  |
|        BertForQuestionAnswering         | 128 |  0.9912  |
|       RobertaForQuestionAnswering       | 128 |  0.9911  |
|                 BigBird                 |  1  |  0.9911  |
|         MegatronBertForCausalLM         | 16  |  0.9903  |
|      MBartForConditionalGeneration      | 16  |  0.9879  |
|           LayoutLMForMaskedLM           | 16  |  0.9876  |
|               GoogleFnet                |  1  |  0.9849  |
|     PLBartForConditionalGeneration      | 16  |  0.9832  |
|               DistillGPT2               |  1  |  0.9832  |
|          MobileBertForMaskedLM          | 32  |  0.9793  |
|            XLNetLMHeadModel             | 32  |  0.9764  |
|          AllenaiLongformerBase          |  1  |  0.9739  |
|             XGLMForCausalLM             |  8  |  0.9707  |
|    MegatronBertForQuestionAnswering     | 16  |   0.96   |
|     M2M100ForConditionalGeneration      |  8  |  0.9463  |
|            YituTechConvBert             |  1  |  0.9448  |
|                CamemBert                |  1  |  0.9319  |
|     MobileBertForQuestionAnswering      | 64  |  0.8914  |
|       DebertaForQuestionAnswering       |  8  |  0.805   |
|                 T5Small                 |  1  |  0.7752  |
|       MT5ForConditionalGeneration       |  8  |  0.7396  |
+-----------------------------------------+-----+----------+

timm_models suite with float32 precision

see more

Performance speedup

+---------------------------------+-----+----------+
|              name               | bs  | inductor |
+---------------------------------+-----+----------+
|          pnasnet5large          | 16  |  1.5208  |
|        gluon_xception65         | 32  |  1.4163  |
|           resnest101e           | 64  |  1.3082  |
|          inception_v3           | 128 |  1.3007  |
|        adv_inception_v3         | 128 |  1.286   |
|       gluon_inception_v3        | 128 |  1.2831  |
|        res2net101_26w_4s        | 64  |  1.2057  |
|           volo_d1_224           | 64  |  1.2056  |
|           fbnetc_100            | 128 |  1.2011  |
|        res2net50_14w_8s         | 128 |  1.1988  |
|          ghostnet_100           | 128 |  1.1984  |
|           res2next50            | 128 |  1.198   |
|           mnasnet_100           | 128 |  1.1891  |
|          spnasnet_100           | 128 |  1.1889  |
|      mobilenetv3_large_100      | 128 |  1.1738  |
|        ese_vovnet19b_dw         | 128 |  1.1707  |
|            hrnet_w18            | 128 |  1.1678  |
|             dpn107              | 32  |  1.1573  |
|            lcnet_050            | 128 |  1.1561  |
|            repvgg_a2            | 128 |  1.1224  |
|            gernet_l             | 128 |  1.1206  |
|            fbnetv3_b            | 128 |  1.1005  |
|         visformer_small         | 128 |  1.0995  |
|           selecsls42b           | 128 |  1.0811  |
|         mobilenetv2_100         | 128 |  1.0765  |
|           regnety_002           | 128 |  1.0605  |
|          cspdarknet53           | 64  |  1.0587  |
|           tf_mixnet_l           | 128 |  1.012   |
|      xcit_large_24_p8_224       |  5  |  1.0046  |
|          gmixer_24_224          | 128 |  0.9926  |
|            mixnet_l             | 128 |  0.9656  |
|          cait_m36_384           |  4  |  0.9542  |
|             dla102              | 128 |  0.9532  |
|           dm_nfnet_f0           | 128 |  0.9113  |
|  swin_base_patch4_window7_224   | 64  |  0.8918  |
|      beit_base_patch16_224      | 64  |  0.8875  |
|          convnext_base          | 64  |  0.8669  |
|         poolformer_m36          | 64  |  0.8643  |
| deit_base_distilled_patch16_224 | 64  |  0.8496  |
|      vit_base_patch16_224       | 64  |  0.8432  |
|            nfnet_l0             | 128 |  0.8416  |
|           convit_base           | 64  |  0.8362  |
|          resmlp_12_224          | 128 |  0.8293  |
|          jx_nest_base           | 32  |  0.8262  |
|         coat_lite_mini          | 128 |  0.822   |
|           rexnet_100            | 128 |  0.8213  |
|          mixer_b16_224          | 128 |  0.8027  |
|        tnt_s_patch16_224        | 128 |  0.8001  |
|            tinynet_a            | 128 |   0.8    |
|       tf_efficientnet_b0        | 128 |  0.7965  |
|            pit_b_224            | 64  |  0.7856  |
|           mobilevit_s           | 64  |  0.7611  |
|        twins_pcpvt_base         | 64  |  0.7438  |
|         crossvit_9_240          | 128 |  0.7234  |
|          gmlp_s16_224           | 128 |  0.6081  |
|     swsl_resnext101_32x16d      | 32  |  0.0721  |
|        sebotnet33ts_256         |  0  |   0.0    |
|        convmixer_768_32         |  0  |   0.0    |
|       eca_botnext26ts_256       |  0  |   0.0    |
|          botnet26t_256          |  0  |   0.0    |
|        eca_halonext26ts         |  0  |   0.0    |
+---------------------------------+-----+----------+

Accuracy

+---------------------------------+----+---------------+
|              name               | bs |   inductor    |
+---------------------------------+----+---------------+
|        adv_inception_v3         | 2  |     pass      |
|             dpn107              | 2  |     pass      |
|      beit_base_patch16_224      | 2  |     pass      |
|          mixer_b16_224          | 2  |     pass      |
|        ese_vovnet19b_dw         | 2  |     pass      |
|         coat_lite_mini          | 2  |     pass      |
|           convit_base           | 2  |     pass      |
|          convnext_base          | 2  |     pass      |
|         crossvit_9_240          | 2  |     pass      |
|          cspdarknet53           | 2  |     pass      |
| deit_base_distilled_patch16_224 | 2  |     pass      |
|             dla102              | 2  |     pass      |
|           dm_nfnet_f0           | 2  |     pass      |
|            lcnet_050            | 2  |     pass      |
|           volo_d1_224           | 2  |     pass      |
|      xcit_large_24_p8_224       | 2  |     pass      |
|           fbnetc_100            | 2  |     pass      |
|        gluon_xception65         | 2  |     pass      |
|          jx_nest_base           | 2  |     pass      |
|          inception_v3           | 2  |     pass      |
|            hrnet_w18            | 2  |     pass      |
|          gmlp_s16_224           | 2  |     pass      |
|          gmixer_24_224          | 2  |     pass      |
|       gluon_inception_v3        | 2  |     pass      |
|            gernet_l             | 2  |     pass      |
|            fbnetv3_b            | 2  |     pass      |
|           mnasnet_100           | 2  |     pass      |
|            mixnet_l             | 2  |     pass      |
|      vit_base_patch16_224       | 2  |     pass      |
|           res2next50            | 2  |     pass      |
|         mobilenetv2_100         | 2  |     pass      |
|      mobilenetv3_large_100      | 2  |     pass      |
|           mobilevit_s           | 2  |     pass      |
|            nfnet_l0             | 2  |     pass      |
|            pit_b_224            | 2  |     pass      |
|          pnasnet5large          | 2  |     pass      |
|         poolformer_m36          | 2  |     pass      |
|           regnety_002           | 2  |     pass      |
|            repvgg_a2            | 2  |     pass      |
|         visformer_small         | 2  |     pass      |
|        res2net50_14w_8s         | 2  |     pass      |
|        res2net101_26w_4s        | 2  |     pass      |
|          resmlp_12_224          | 2  |     pass      |
|       tf_efficientnet_b0        | 2  |     pass      |
|        twins_pcpvt_base         | 2  |     pass      |
|        tnt_s_patch16_224        | 2  |     pass      |
|           resnest101e           | 2  |     pass      |
|           tf_mixnet_l           | 2  |     pass      |
|            tinynet_a            | 2  |     pass      |
|     swsl_resnext101_32x16d      | 2  |     pass      |
|  swin_base_patch4_window7_224   | 2  |     pass      |
|          spnasnet_100           | 2  |     pass      |
|           selecsls42b           | 2  |     pass      |
|           rexnet_100            | 2  |     pass      |
|        eca_halonext26ts         | 2  |  fail_to_run  |
|        convmixer_768_32         | 2  |  fail_to_run  |
|        sebotnet33ts_256         | 2  |  fail_to_run  |
|          botnet26t_256          | 2  |  fail_to_run  |
|       eca_botnext26ts_256       | 2  |  fail_to_run  |
|          ghostnet_100           | 2  | fail_accuracy |
|          cait_m36_384           | 2  | fail_accuracy |
+---------------------------------+----+---------------+

Compilation latency (sec)

+---------------------------------+-----+----------+
|              name               | bs  | inductor |
+---------------------------------+-----+----------+
|     swsl_resnext101_32x16d      | 32  | 116.2647 |
|          pnasnet5large          | 16  | 34.3568  |
|            hrnet_w18            | 128 | 33.1729  |
|  swin_base_patch4_window7_224   | 64  | 30.3657  |
|          cait_m36_384           |  4  |  26.113  |
|       tf_efficientnet_b0        | 128 | 26.0213  |
|           dm_nfnet_f0           | 128 | 25.8731  |
|           tf_mixnet_l           | 128 | 25.1199  |
|        twins_pcpvt_base         | 64  | 24.4181  |
|             dla102              | 128 | 24.3209  |
|           mobilevit_s           | 64  | 24.2889  |
|         poolformer_m36          | 64  | 23.7559  |
|            mixnet_l             | 128 | 22.9678  |
|        res2net50_14w_8s         | 128 | 22.6444  |
|      xcit_large_24_p8_224       |  5  | 22.5081  |
|        tnt_s_patch16_224        | 128 |  22.374  |
|           resnest101e           | 64  | 22.0767  |
|            nfnet_l0             | 128 | 21.8411  |
|            fbnetv3_b            | 128 | 21.3069  |
|        res2net101_26w_4s        | 64  |  21.205  |
|             dpn107              | 32  | 21.1366  |
|           rexnet_100            | 128 | 20.9301  |
|          gmlp_s16_224           | 128 | 20.5459  |
|          jx_nest_base           | 32  | 19.9281  |
|          convnext_base          | 64  |  18.721  |
|           volo_d1_224           | 64  | 18.2821  |
|         crossvit_9_240          | 128 | 18.1307  |
|           convit_base           | 64  | 17.8616  |
|            tinynet_a            | 128 | 17.5486  |
|           res2next50            | 128 | 17.3809  |
|            pit_b_224            | 64  | 17.1044  |
|         coat_lite_mini          | 128 | 17.0934  |
|          cspdarknet53           | 64  |  16.315  |
|       gluon_inception_v3        | 128 |  16.027  |
|          inception_v3           | 128 | 15.9994  |
|        adv_inception_v3         | 128 | 15.9628  |
|        gluon_xception65         | 32  | 15.8657  |
|          gmixer_24_224          | 128 | 15.5962  |
|          mixer_b16_224          | 128 | 15.3517  |
|          ghostnet_100           | 128 | 15.1605  |
|         visformer_small         | 128 | 14.9027  |
|      beit_base_patch16_224      | 64  | 14.7057  |
| deit_base_distilled_patch16_224 | 64  | 14.5395  |
|         mobilenetv2_100         | 128 | 14.4137  |
|      mobilenetv3_large_100      | 128 | 14.2885  |
|      vit_base_patch16_224       | 64  | 14.2558  |
|           fbnetc_100            | 128 | 14.1446  |
|          spnasnet_100           | 128 | 13.8994  |
|            gernet_l             | 128 | 13.4121  |
|           mnasnet_100           | 128 | 12.7666  |
|           regnety_002           | 128 | 12.5997  |
|            repvgg_a2            | 128 | 12.5894  |
|        ese_vovnet19b_dw         | 128 | 12.1089  |
|           selecsls42b           | 128 | 12.0156  |
|          resmlp_12_224          | 128 | 11.5499  |
|            lcnet_050            | 128 | 10.6792  |
|          botnet26t_256          |  0  |   nan    |
|        convmixer_768_32         |  0  |   nan    |
|       eca_botnext26ts_256       |  0  |   nan    |
|        eca_halonext26ts         |  0  |   nan    |
|        sebotnet33ts_256         |  0  |   nan    |
+---------------------------------+-----+----------+

Peak Memory Compression Ratio

+---------------------------------+-----+----------+
|              name               | bs  | inductor |
+---------------------------------+-----+----------+
|        adv_inception_v3         | 128 |  0.9973  |
|          mixer_b16_224          | 128 |  0.9973  |
|           resnest101e           | 64  |  0.997   |
|        ese_vovnet19b_dw         | 128 |  0.997   |
|          inception_v3           | 128 |  0.9969  |
|             dla102              | 128 |  0.9969  |
|           dm_nfnet_f0           | 128 |  0.9968  |
|           tf_mixnet_l           | 128 |  0.9967  |
|         coat_lite_mini          | 128 |  0.9966  |
|            mixnet_l             | 128 |  0.9964  |
| deit_base_distilled_patch16_224 | 64  |  0.9964  |
|           convit_base           | 64  |  0.9964  |
|          cspdarknet53           | 64  |  0.9963  |
|      vit_base_patch16_224       | 64  |  0.9963  |
|          gmixer_24_224          | 128 |  0.9962  |
|      beit_base_patch16_224      | 64  |  0.9962  |
|           selecsls42b           | 128 |  0.9962  |
|        gluon_xception65         | 32  |  0.9961  |
|            gernet_l             | 128 |  0.9961  |
|          resmlp_12_224          | 128 |  0.996   |
|            fbnetv3_b            | 128 |  0.996   |
|          convnext_base          | 64  |  0.9956  |
|           mobilevit_s           | 64  |  0.9954  |
|          gmlp_s16_224           | 128 |  0.9954  |
|            pit_b_224            | 64  |  0.9954  |
|       tf_efficientnet_b0        | 128 |  0.9952  |
|            tinynet_a            | 128 |  0.9949  |
|           res2next50            | 128 |  0.9947  |
|           fbnetc_100            | 128 |  0.9946  |
|           mnasnet_100           | 128 |  0.9944  |
|          spnasnet_100           | 128 |  0.9943  |
|            hrnet_w18            | 128 |  0.9943  |
|           rexnet_100            | 128 |  0.9942  |
|         visformer_small         | 128 |  0.9941  |
|        tnt_s_patch16_224        | 128 |  0.9941  |
|            repvgg_a2            | 128 |  0.9941  |
|         mobilenetv2_100         | 128 |  0.9941  |
|      mobilenetv3_large_100      | 128 |  0.994   |
|       gluon_inception_v3        | 128 |  0.9939  |
|             dpn107              | 32  |  0.9937  |
|          cait_m36_384           |  4  |  0.9935  |
|            nfnet_l0             | 128 |  0.9934  |
|        res2net50_14w_8s         | 128 |  0.9933  |
|         poolformer_m36          | 64  |  0.9931  |
|  swin_base_patch4_window7_224   | 64  |  0.9929  |
|          jx_nest_base           | 32  |  0.9925  |
|        res2net101_26w_4s        | 64  |  0.9924  |
|          ghostnet_100           | 128 |  0.9923  |
|        twins_pcpvt_base         | 64  |  0.9915  |
|         crossvit_9_240          | 128 |  0.991   |
|          pnasnet5large          | 16  |  0.9904  |
|           volo_d1_224           | 64  |  0.9903  |
|      xcit_large_24_p8_224       |  5  |  0.9901  |
|     swsl_resnext101_32x16d      | 32  |  0.9898  |
|            lcnet_050            | 128 |  0.9883  |
|           regnety_002           | 128 |  0.9879  |
|          botnet26t_256          |  0  |   nan    |
|        convmixer_768_32         |  0  |   nan    |
|       eca_botnext26ts_256       |  0  |   nan    |
|        eca_halonext26ts         |  0  |   nan    |
|        sebotnet33ts_256         |  0  |   nan    |
+---------------------------------+-----+----------+

[chuanqi129]

Performance Dashboard for float32 precision -- Single-core Single-thread (2022-11-09 nightly release)

Executive Summary

see more

We evaluate different backends across three benchmark suites - torchbench, huggingface and timm. We run these experiments on Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz. Each experiment runs one iteration of forward pass. For accuracy, we check the numerical correctness of forward pass outputs by comparing with native pytorch. We measure speedup by normalizing against the performance of native pytorch. We report mean compilation latency numbers and peak memory footprint reduction ratio.

Caveats

1. Batch size has been reduced to workaround OOM errors. Work is in progress to reduce peak memory footprint.
2. Experiments do not cover dynamic shapes.
3. Experimental setup does not have optimizer.

SW information

SW	Nightly commit	Master/Main commit
Pytorch	3b29687	a7420d2
Torchbench	/	022dfe3
torchaudio	4b10b6a	74f9a89
torchtext	71e4561	c047efe
torchvision	797e1ac	ffd5a56

HW information

Item	Value
Manufacturer	Amazon EC2
Product Name	c6i.16xlarge
CPU Model	Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz
Installed Memory	128GB (1x128GB DDR4 3200 MT/s [Unknown])
OS	Ubuntu 20.04.5 LTS
Kernel	5.15.0-1022-aws
Microcode	0xd000331
GCC	gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
GLIBC	ldd (Ubuntu GLIBC 2.31-0ubuntu9.9) 2.31
Binutils	GNU ld (GNU Binutils for Ubuntu) 2.34
Python	Python 3.8.13
OpenSSL	OpenSSL 1.1.1s 1 Nov 2022

To measure performance, compilation latency and memory footprint reduction, we remove the models that fail accuracy checks.

Passrate

+----------+-------------+-------------+-------------+
| Compiler | torchbench  | huggingface | timm_models |
+----------+-------------+-------------+-------------+
| inductor | 100%, 55/55 | 100%, 44/44 | 92%, 56/61  |
+----------+-------------+-------------+-------------+

Geometric mean speedup

+----------+------------+-------------+-------------+
| Compiler | torchbench | huggingface | timm_models |
+----------+------------+-------------+-------------+
| inductor |   1.05x    |    1.00x    |    1.08x    |
+----------+------------+-------------+-------------+

Mean compilation time (seconds)

+----------+------------+-------------+-------------+
| Compiler | torchbench | huggingface | timm_models |
+----------+------------+-------------+-------------+
| inductor |   20.31    |    21.37    |    24.75    |
+----------+------------+-------------+-------------+

Peak memory footprint compression ratio (higher is better)

+----------+------------+-------------+-------------+
| Compiler | torchbench | huggingface | timm_models |
+----------+------------+-------------+-------------+
| inductor |   1.10x    |    1.29x    |    1.07x    |
+----------+------------+-------------+-------------+

torchbench suite with float32 precision

see more

Performance speedup

+-----------------------------------+-----+----------+
|               name                | bs  | inductor |
+-----------------------------------+-----+----------+
|        shufflenet_v2_x1_0         |  1  |  1.4852  |
|           squeezenet1_1           |  1  |  1.2136  |
|           hf_Longformer           |  1  |  1.1919  |
|   pytorch_CycleGAN_and_pix2pix    |  1  |  1.1811  |
|            timm_nfnet             |  1  |  1.1635  |
|       functorch_dp_cifar10        |  1  |  1.1517  |
|             resnet18              |  1  |  1.1458  |
|           pytorch_unet            |  1  |  1.1379  |
|            timm_vovnet            |  1  |  1.1126  |
| attention_is_all_you_need_pytorch |  1  |  1.1067  |
|     detectron2_fcos_r_50_fpn      |  1  |  1.094   |
|            Super_SloMo            |  1  |  1.0921  |
|               vgg16               |  1  |  1.0901  |
|              alexnet              |  1  |  1.0825  |
|                drq                |  1  |  1.0759  |
|            timm_regnet            |  1  |  1.0654  |
|        Background_Matting         |  1  |  1.0627  |
|          LearningToPaint          |  1  |  1.0595  |
|         soft_actor_critic         | 256 |  1.0328  |
|               dcgan               |  1  |  1.032   |
|               dlrm                |  1  |  1.0286  |
|           timm_resnest            |  1  |  1.0247  |
|          pytorch_stargan          | 16  |  1.0146  |
|            densenet121            |  1  |  1.001   |
|              demucs               |  1  |  0.9994  |
|            tts_angular            |  1  |  0.998   |
|      resnet50_quantized_qat       |  1  |  0.9967  |
|            hf_BigBird             |  1  |  0.9922  |
|    mobilenet_v2_quantized_qat     |  1  |  0.9921  |
|      nvidia_deeprecommender       |  1  |  0.984   |
|             resnet152             |  1  |  0.9384  |
|        speech_transformer         |  1  |  0.9359  |
|           mobilenet_v2            |  1  |  0.9322  |
|            mnasnet1_0             |  1  |  0.9098  |
|          vision_maskrcnn          |  1  |  0.8935  |
|           BERT_pytorch            |  1  |  0.891   |
|              yolov3               |  1  |  0.8903  |
|              hf_GPT2              |  1  |  0.8772  |
|             resnet50              |  1  |  0.8629  |
|   timm_vision_transformer_large   |  1  |  0.8603  |
|             hf_Albert             |  1  |  0.8272  |
|            hf_Reformer            |  1  |  0.8229  |
|        mobilenet_v3_large         |  1  |  0.8028  |
|            hf_T5_large            |  1  |  0.7941  |
|           hf_DistilBert           |  1  |  0.7891  |
|           lennard_jones           |  1  |  0.7829  |
|           fastNLP_Bert            |  1  |  0.7693  |
|              hf_Bert              |  1  |  0.7587  |
|               hf_T5               |  1  |  0.7556  |
|           hf_GPT2_large           |  1  |  0.7493  |
|          resnext50_32x4d          |  1  |  0.7484  |
|              hf_Bart              |  1  |  0.742   |
|      timm_vision_transformer      |  1  |  0.6809  |
|            hf_T5_base             |  1  |   0.67   |
|         timm_efficientnet         |  1  |  0.5005  |
+-----------------------------------+-----+----------+

Accuracy

+-----------------------------------+-----+------------------+
|               name                | bs  |     inductor     |
+-----------------------------------+-----+------------------+
|            hf_T5_large            |  1  | pass_due_to_skip |
|           hf_GPT2_large           |  1  | pass_due_to_skip |
|   timm_vision_transformer_large   |  1  | pass_due_to_skip |
|              yolov3               |  1  |       pass       |
|                drq                |  1  |       pass       |
|           lennard_jones           |  1  |       pass       |
|          LearningToPaint          |  1  |       pass       |
|            Super_SloMo            |  1  |       pass       |
|              alexnet              |  1  |       pass       |
| attention_is_all_you_need_pytorch |  1  |       pass       |
|               dcgan               |  1  |       pass       |
|              demucs               |  1  |       pass       |
|            densenet121            |  1  |       pass       |
|     detectron2_fcos_r_50_fpn      |  1  |       pass       |
|               dlrm                |  1  |       pass       |
|           fastNLP_Bert            |  1  |       pass       |
|            hf_T5_base             |  1  |       pass       |
|        Background_Matting         |  1  |       pass       |
|             hf_Albert             |  1  |       pass       |
|              hf_Bart              |  1  |       pass       |
|              hf_Bert              |  1  |       pass       |
|            hf_BigBird             |  1  |       pass       |
|           hf_DistilBert           |  1  |       pass       |
|              hf_GPT2              |  1  |       pass       |
|           hf_Longformer           |  1  |       pass       |
|            hf_Reformer            |  1  |       pass       |
|               hf_T5               |  1  |       pass       |
|       functorch_dp_cifar10        |  1  |       pass       |
|    mobilenet_v2_quantized_qat     |  1  |       pass       |
|            mnasnet1_0             |  1  |       pass       |
|               vgg16               |  1  |       pass       |
|           BERT_pytorch            |  1  |       pass       |
|          resnext50_32x4d          |  1  |       pass       |
|        mobilenet_v3_large         |  1  |       pass       |
|      nvidia_deeprecommender       |  1  |       pass       |
|   pytorch_CycleGAN_and_pix2pix    |  1  |       pass       |
|          pytorch_stargan          | 16  |       pass       |
|           pytorch_unet            |  1  |       pass       |
|             resnet152             |  1  |       pass       |
|             resnet18              |  1  |       pass       |
|             resnet50              |  1  |       pass       |
|      resnet50_quantized_qat       |  1  |       pass       |
|        shufflenet_v2_x1_0         |  1  |       pass       |
|           mobilenet_v2            |  1  |       pass       |
|         soft_actor_critic         | 256 |       pass       |
|        speech_transformer         |  1  |       pass       |
|           squeezenet1_1           |  1  |       pass       |
|         timm_efficientnet         |  1  |       pass       |
|            timm_nfnet             |  1  |       pass       |
|            timm_regnet            |  1  |       pass       |
|           timm_resnest            |  1  |       pass       |
|      timm_vision_transformer      |  1  |       pass       |
|            timm_vovnet            |  1  |       pass       |
|            tts_angular            |  1  |       pass       |
|          vision_maskrcnn          |  0  |      0.0000      |
+-----------------------------------+-----+------------------+

Compilation latency (sec)

+-----------------------------------+-----+----------+
|               name                | bs  | inductor |
+-----------------------------------+-----+----------+
|            hf_T5_base             |  1  | 81.7092  |
|          vision_maskrcnn          |  1  | 72.4651  |
|     detectron2_fcos_r_50_fpn      |  1  |  69.401  |
|           hf_GPT2_large           |  1  | 59.9626  |
|            hf_T5_large            |  1  | 54.6313  |
|            densenet121            |  1  | 33.7722  |
|              yolov3               |  1  |  28.69   |
|   timm_vision_transformer_large   |  1  | 28.2311  |
|            timm_nfnet             |  1  | 26.8436  |
|           hf_Longformer           |  1  |  26.637  |
|            Super_SloMo            |  1  | 26.4081  |
|            hf_BigBird             |  1  | 24.8659  |
|         timm_efficientnet         |  1  | 24.7895  |
|        Background_Matting         |  1  | 24.5048  |
|           BERT_pytorch            |  1  | 24.3098  |
|             resnet152             |  1  | 21.2911  |
|            timm_regnet            |  1  |  20.07   |
|        speech_transformer         |  1  | 19.8299  |
|               hf_T5               |  1  | 19.5154  |
|        mobilenet_v3_large         |  1  | 19.4807  |
|            hf_Reformer            |  1  | 18.6868  |
|              hf_Bart              |  1  | 18.0463  |
|           pytorch_unet            |  1  | 17.9894  |
|           fastNLP_Bert            |  1  | 17.0308  |
|          resnext50_32x4d          |  1  | 16.6334  |
|           mobilenet_v2            |  1  | 16.5489  |
|             resnet50              |  1  | 16.2093  |
|              hf_GPT2              |  1  | 16.1715  |
|              hf_Bert              |  1  |  15.522  |
|            mnasnet1_0             |  1  | 14.8517  |
|           timm_resnest            |  1  | 14.7064  |
|            timm_vovnet            |  1  | 14.6828  |
|             hf_Albert             |  1  |  14.343  |
|        shufflenet_v2_x1_0         |  1  | 14.1955  |
| attention_is_all_you_need_pytorch |  1  | 14.1544  |
|       functorch_dp_cifar10        |  1  | 13.9881  |
|      timm_vision_transformer      |  1  | 13.7393  |
|           hf_DistilBert           |  1  | 13.1191  |
|          pytorch_stargan          | 16  | 12.5482  |
|   pytorch_CycleGAN_and_pix2pix    |  1  | 11.5582  |
|           squeezenet1_1           |  1  | 11.2071  |
|          LearningToPaint          |  1  | 10.6182  |
|                drq                |  1  |  9.9081  |
|               vgg16               |  1  |  9.7425  |
|             resnet18              |  1  |  9.7095  |
|               dlrm                |  1  |  9.5449  |
|      nvidia_deeprecommender       |  1  |  9.441   |
|              alexnet              |  1  |   9.03   |
|            tts_angular            |  1  |  8.247   |
|         soft_actor_critic         | 256 |  8.2421  |
|           lennard_jones           |  1  |  7.8457  |
|              demucs               |  1  |  1.1898  |
|               dcgan               |  1  |  0.1695  |
|    mobilenet_v2_quantized_qat     |  1  |  0.0748  |
|      resnet50_quantized_qat       |  1  |  0.0579  |
+-----------------------------------+-----+----------+

Peak Memory Compression Ratio

+-----------------------------------+-----+----------+
|               name                | bs  | inductor |
+-----------------------------------+-----+----------+
|           pytorch_unet            |  1  |  2.1274  |
|             hf_Albert             |  1  |  2.0308  |
|            hf_BigBird             |  1  |  1.7822  |
|            Super_SloMo            |  1  |  1.7473  |
|            hf_T5_large            |  1  |  1.4919  |
|          vision_maskrcnn          |  1  |  1.4585  |
|        speech_transformer         |  1  |  1.4474  |
|   pytorch_CycleGAN_and_pix2pix    |  1  |  1.4467  |
|           fastNLP_Bert            |  1  |  1.359   |
|            hf_T5_base             |  1  |  1.3165  |
|               hf_T5               |  1  |  1.2507  |
|            timm_nfnet             |  1  |  1.1479  |
|            timm_regnet            |  1  |  1.1406  |
|   timm_vision_transformer_large   |  1  |  1.1151  |
|         soft_actor_critic         | 256 |  1.1148  |
|              hf_GPT2              |  1  |  1.1111  |
|         timm_efficientnet         |  1  |  1.0556  |
|            timm_vovnet            |  1  |  1.0511  |
|           mobilenet_v2            |  1  |  1.0508  |
|      timm_vision_transformer      |  1  |  1.0422  |
|              yolov3               |  1  |  1.0148  |
|           hf_GPT2_large           |  1  |  1.0104  |
|               dlrm                |  1  |  1.0008  |
|              demucs               |  1  |  0.9986  |
|      nvidia_deeprecommender       |  1  |  0.9973  |
|      resnet50_quantized_qat       |  1  |  0.9953  |
|        Background_Matting         |  1  |  0.9952  |
|            tts_angular            |  1  |  0.9931  |
|    mobilenet_v2_quantized_qat     |  1  |  0.9927  |
|          pytorch_stargan          | 16  |  0.9886  |
|           lennard_jones           |  1  |  0.9833  |
|        mobilenet_v3_large         |  1  |  0.9811  |
|            mnasnet1_0             |  1  |  0.9796  |
|     detectron2_fcos_r_50_fpn      |  1  |  0.9783  |
|                drq                |  1  |  0.9753  |
|               vgg16               |  1  |  0.9738  |
| attention_is_all_you_need_pytorch |  1  |  0.9696  |
|           squeezenet1_1           |  1  |  0.9683  |
|              alexnet              |  1  |  0.9671  |
|           timm_resnest            |  1  |  0.9604  |
|        shufflenet_v2_x1_0         |  1  |  0.9602  |
|           hf_Longformer           |  1  |  0.9518  |
|       functorch_dp_cifar10        |  1  |  0.9511  |
|               dcgan               |  1  |  0.9463  |
|          LearningToPaint          |  1  |  0.9427  |
|             resnet50              |  1  |  0.9418  |
|              hf_Bart              |  1  |  0.9408  |
|           BERT_pytorch            |  1  |  0.9357  |
|              hf_Bert              |  1  |  0.9295  |
|             resnet18              |  1  |  0.9199  |
|             resnet152             |  1  |  0.9161  |
|           hf_DistilBert           |  1  |  0.8963  |
|          resnext50_32x4d          |  1  |  0.8052  |
|            hf_Reformer            |  1  |  0.7742  |
|            densenet121            |  1  |  0.7664  |
+-----------------------------------+-----+----------+

huggingface suite with float32 precision

see more

Performance speedup

+-----------------------------------------+----+----------+
|                  name                   | bs | inductor |
+-----------------------------------------+----+----------+
|            XLNetLMHeadModel             | 1  |  1.1042  |
|     MobileBertForQuestionAnswering      | 1  |  1.0831  |
|       AlbertForQuestionAnswering        | 1  |  0.9533  |
|            AlbertForMaskedLM            | 1  |  0.9478  |
|          MobileBertForMaskedLM          | 1  |  0.9301  |
|     M2M100ForConditionalGeneration      | 1  |  0.8985  |
|                 BigBird                 | 1  |  0.8978  |
|            YituTechConvBert             | 1  |  0.8934  |
|             OPTForCausalLM              | 1  |  0.8727  |
|       DebertaForQuestionAnswering       | 1  |  0.8613  |
|               GoogleFnet                | 1  |  0.8548  |
|    MegatronBertForQuestionAnswering     | 1  |  0.8545  |
|         MegatronBertForCausalLM         | 1  |  0.8449  |
|      MBartForConditionalGeneration      | 1  |  0.8443  |
|     PegasusForConditionalGeneration     | 1  |  0.843   |
|     DistilBertForQuestionAnswering      | 1  |  0.8402  |
|        BertForQuestionAnswering         | 1  |  0.8328  |
|            TrOCRForCausalLM             | 1  |  0.8311  |
|            MBartForCausalLM             | 1  |  0.8303  |
|           PegasusForCausalLM            | 1  |  0.8296  |
|       RobertaForQuestionAnswering       | 1  |  0.8235  |
|          AllenaiLongformerBase          | 1  |  0.8232  |
|           DebertaForMaskedLM            | 1  |  0.8209  |
|             XGLMForCausalLM             | 1  |  0.8092  |
|         Speech2Text2ForCausalLM         | 1  |  0.8035  |
|     PLBartForConditionalGeneration      | 1  |  0.8028  |
|             BertForMaskedLM             | 1  |  0.7952  |
|           RobertaForCausalLM            | 1  |  0.7938  |
|          DistilBertForMaskedLM          | 1  |  0.7847  |
|            PLBartForCausalLM            | 1  |  0.7766  |
|               DistillGPT2               | 1  |  0.7334  |
|           LayoutLMForMaskedLM           | 1  |  0.7261  |
|    LayoutLMForSequenceClassification    | 1  |  0.7245  |
| BlenderbotSmallForConditionalGeneration | 1  |  0.7184  |
|                CamemBert                | 1  |  0.7103  |
|       BlenderbotSmallForCausalLM        | 1  |  0.6951  |
|       MT5ForConditionalGeneration       | 1  |  0.6884  |
|             BartForCausalLM             | 1  |  0.6786  |
|      GPT2ForSequenceClassification      | 1  |  0.6739  |
|      BartForConditionalGeneration       | 1  |  0.6451  |
|       T5ForConditionalGeneration        | 1  |  0.632   |
|                 T5Small                 | 1  |  0.6313  |
|       ElectraForQuestionAnswering       | 1  |  0.5732  |
|           ElectraForCausalLM            | 1  |  0.5332  |
+-----------------------------------------+----+----------+

Accuracy

+-----------------------------------------+----+----------+
|                  name                   | bs | inductor |
+-----------------------------------------+----+----------+
|            AlbertForMaskedLM            | 1  |   pass   |
|       AlbertForQuestionAnswering        | 1  |   pass   |
|                CamemBert                | 1  |   pass   |
|          AllenaiLongformerBase          | 1  |   pass   |
|             BartForCausalLM             | 1  |   pass   |
|      BartForConditionalGeneration       | 1  |   pass   |
|             BertForMaskedLM             | 1  |   pass   |
|        BertForQuestionAnswering         | 1  |   pass   |
|                 BigBird                 | 1  |   pass   |
|       BlenderbotSmallForCausalLM        | 1  |   pass   |
| BlenderbotSmallForConditionalGeneration | 1  |   pass   |
|           DebertaForMaskedLM            | 1  |   pass   |
|           LayoutLMForMaskedLM           | 1  |   pass   |
|       DebertaForQuestionAnswering       | 1  |   pass   |
|          DistilBertForMaskedLM          | 1  |   pass   |
|     DistilBertForQuestionAnswering      | 1  |   pass   |
|               DistillGPT2               | 1  |   pass   |
|           ElectraForCausalLM            | 1  |   pass   |
|       ElectraForQuestionAnswering       | 1  |   pass   |
|      GPT2ForSequenceClassification      | 1  |   pass   |
|               GoogleFnet                | 1  |   pass   |
|    LayoutLMForSequenceClassification    | 1  |   pass   |
|     M2M100ForConditionalGeneration      | 1  |   pass   |
|            MBartForCausalLM             | 1  |   pass   |
|     PLBartForConditionalGeneration      | 1  |   pass   |
|      MBartForConditionalGeneration      | 1  |   pass   |
|       MT5ForConditionalGeneration       | 1  |   pass   |
|         MegatronBertForCausalLM         | 1  |   pass   |
|    MegatronBertForQuestionAnswering     | 1  |   pass   |
|          MobileBertForMaskedLM          | 1  |   pass   |
|     MobileBertForQuestionAnswering      | 1  |   pass   |
|             OPTForCausalLM              | 1  |   pass   |
|            PLBartForCausalLM            | 1  |   pass   |
|           PegasusForCausalLM            | 1  |   pass   |
|            XLNetLMHeadModel             | 1  |   pass   |
|     PegasusForConditionalGeneration     | 1  |   pass   |
|           RobertaForCausalLM            | 1  |   pass   |
|       RobertaForQuestionAnswering       | 1  |   pass   |
|         Speech2Text2ForCausalLM         | 1  |   pass   |
|       T5ForConditionalGeneration        | 1  |   pass   |
|                 T5Small                 | 1  |   pass   |
|            TrOCRForCausalLM             | 1  |   pass   |
|             XGLMForCausalLM             | 1  |   pass   |
|            YituTechConvBert             | 1  |   pass   |
+-----------------------------------------+----+----------+

Compilation latency (sec)

+-----------------------------------------+----+----------+
|                  name                   | bs | inductor |
+-----------------------------------------+----+----------+
|      BartForConditionalGeneration       | 1  | 48.1056  |
|       AlbertForQuestionAnswering        | 1  | 34.3286  |
|            AlbertForMaskedLM            | 1  | 33.2098  |
|          AllenaiLongformerBase          | 1  | 32.9485  |
|       MT5ForConditionalGeneration       | 1  | 32.3667  |
|     M2M100ForConditionalGeneration      | 1  | 26.2587  |
|          MobileBertForMaskedLM          | 1  | 26.1128  |
|                 BigBird                 | 1  | 25.7282  |
|             BartForCausalLM             | 1  | 25.6498  |
|     MobileBertForQuestionAnswering      | 1  | 25.6126  |
|     PegasusForConditionalGeneration     | 1  | 25.5287  |
|       T5ForConditionalGeneration        | 1  | 24.7543  |
|           DebertaForMaskedLM            | 1  | 24.7295  |
|            XLNetLMHeadModel             | 1  |  24.657  |
|      MBartForConditionalGeneration      | 1  | 24.5472  |
|                 T5Small                 | 1  | 24.3507  |
|             XGLMForCausalLM             | 1  | 23.8144  |
|            TrOCRForCausalLM             | 1  |  23.758  |
|       DebertaForQuestionAnswering       | 1  | 23.4078  |
|          DistilBertForMaskedLM          | 1  | 22.5849  |
|      GPT2ForSequenceClassification      | 1  | 22.0168  |
|         MegatronBertForCausalLM         | 1  | 21.8418  |
|    MegatronBertForQuestionAnswering     | 1  | 21.4816  |
|            YituTechConvBert             | 1  | 19.0803  |
|           PegasusForCausalLM            | 1  | 17.7898  |
| BlenderbotSmallForConditionalGeneration | 1  | 17.4371  |
|             BertForMaskedLM             | 1  | 17.1292  |
|                CamemBert                | 1  | 16.7218  |
|     PLBartForConditionalGeneration      | 1  | 16.6943  |
|            MBartForCausalLM             | 1  | 16.6502  |
|           LayoutLMForMaskedLM           | 1  | 16.5857  |
|        BertForQuestionAnswering         | 1  | 16.3166  |
|    LayoutLMForSequenceClassification    | 1  | 16.0688  |
|             OPTForCausalLM              | 1  | 15.9214  |
|               DistillGPT2               | 1  | 15.0388  |
|           RobertaForCausalLM            | 1  | 14.5262  |
|           ElectraForCausalLM            | 1  | 14.4963  |
|       RobertaForQuestionAnswering       | 1  | 14.2469  |
|       ElectraForQuestionAnswering       | 1  | 13.9396  |
|               GoogleFnet                | 1  | 13.6451  |
|       BlenderbotSmallForCausalLM        | 1  | 12.9301  |
|         Speech2Text2ForCausalLM         | 1  | 12.7357  |
|            PLBartForCausalLM            | 1  | 12.5302  |
|     DistilBertForQuestionAnswering      | 1  | 11.8878  |
+-----------------------------------------+----+----------+

Peak Memory Compression Ratio

+-----------------------------------------+----+----------+
|                  name                   | bs | inductor |
+-----------------------------------------+----+----------+
|            AlbertForMaskedLM            | 1  |  2.6371  |
|       AlbertForQuestionAnswering        | 1  |  2.5568  |
|      BartForConditionalGeneration       | 1  |  2.1208  |
|                 T5Small                 | 1  |  1.9472  |
|       T5ForConditionalGeneration        | 1  |  1.8685  |
|                 BigBird                 | 1  |  1.8542  |
|      GPT2ForSequenceClassification      | 1  |  1.8363  |
|             BartForCausalLM             | 1  |  1.5123  |
|          AllenaiLongformerBase          | 1  |  1.4905  |
|            XLNetLMHeadModel             | 1  |  1.3591  |
|       DebertaForQuestionAnswering       | 1  |  1.3508  |
|                CamemBert                | 1  |  1.3349  |
|           LayoutLMForMaskedLM           | 1  |  1.3255  |
|            YituTechConvBert             | 1  |  1.3106  |
|           DebertaForMaskedLM            | 1  |  1.3088  |
|               GoogleFnet                | 1  |  1.2897  |
|    LayoutLMForSequenceClassification    | 1  |  1.2841  |
|           ElectraForCausalLM            | 1  |  1.264   |
|               DistillGPT2               | 1  |   1.19   |
|       ElectraForQuestionAnswering       | 1  |  1.1781  |
|     M2M100ForConditionalGeneration      | 1  |  1.0959  |
| BlenderbotSmallForConditionalGeneration | 1  |  1.0836  |
|     PegasusForConditionalGeneration     | 1  |  1.0813  |
|      MBartForConditionalGeneration      | 1  |  1.0707  |
|       MT5ForConditionalGeneration       | 1  |  1.0639  |
|             BertForMaskedLM             | 1  |  1.0574  |
|         MegatronBertForCausalLM         | 1  |  1.0557  |
|    MegatronBertForQuestionAnswering     | 1  |  1.0497  |
|           RobertaForCausalLM            | 1  |  1.0444  |
|       BlenderbotSmallForCausalLM        | 1  |  1.0425  |
|            PLBartForCausalLM            | 1  |  1.0422  |
|            TrOCRForCausalLM             | 1  |  1.042   |
|             XGLMForCausalLM             | 1  |  1.0411  |
|             OPTForCausalLM              | 1  |  1.0408  |
|        BertForQuestionAnswering         | 1  |  1.038   |
|     PLBartForConditionalGeneration      | 1  |  1.0358  |
|          DistilBertForMaskedLM          | 1  |  1.0342  |
|       RobertaForQuestionAnswering       | 1  |  1.0332  |
|            MBartForCausalLM             | 1  |  1.0272  |
|     DistilBertForQuestionAnswering      | 1  |  1.0157  |
|           PegasusForCausalLM            | 1  |  1.0102  |
|          MobileBertForMaskedLM          | 1  |  0.9923  |
|         Speech2Text2ForCausalLM         | 1  |  0.9793  |
|     MobileBertForQuestionAnswering      | 1  |  0.9747  |
+-----------------------------------------+----+----------+

timm_models suite with float32 precision

see more

Performance speedup

+---------------------------------+----+----------+
|              name               | bs | inductor |
+---------------------------------+----+----------+
|          pnasnet5large          | 1  |  1.5448  |
|           regnety_002           | 1  |  1.3632  |
|            lcnet_050            | 1  |  1.3565  |
|        ese_vovnet19b_dw         | 1  |  1.3444  |
|          spnasnet_100           | 1  |  1.3401  |
|          inception_v3           | 1  |  1.3239  |
|       gluon_inception_v3        | 1  |  1.3096  |
|        adv_inception_v3         | 1  |  1.3006  |
|           mnasnet_100           | 1  |  1.2928  |
|           fbnetc_100            | 1  |  1.2682  |
|          gmixer_24_224          | 1  |  1.2677  |
|         mobilenetv2_100         | 1  |  1.2517  |
|            fbnetv3_b            | 1  |  1.1728  |
|      mobilenetv3_large_100      | 1  |  1.1453  |
|             dpn107              | 1  |  1.0835  |
|            gernet_l             | 1  |  1.0821  |
|          cspdarknet53           | 1  |  1.0688  |
|        gluon_xception65         | 1  |  1.0532  |
|            repvgg_a2            | 1  |  1.0493  |
|            hrnet_w18            | 1  |  1.0471  |
|          ghostnet_100           | 1  |  0.9966  |
|            nfnet_l0             | 1  |  0.9933  |
|           dm_nfnet_f0           | 1  |  0.9885  |
|           resnest101e           | 1  |  0.9787  |
|           selecsls42b           | 1  |  0.9235  |
|         visformer_small         | 1  |  0.9056  |
|        res2net101_26w_4s        | 1  |  0.8988  |
|      xcit_large_24_p8_224       | 1  |  0.8934  |
|          convnext_base          | 1  |  0.8669  |
|        res2net50_14w_8s         | 1  |  0.8607  |
|      beit_base_patch16_224      | 1  |  0.8547  |
|  swin_base_patch4_window7_224   | 1  |  0.8027  |
|           res2next50            | 1  |   0.79   |
| deit_base_distilled_patch16_224 | 1  |  0.7887  |
|           volo_d1_224           | 1  |  0.7649  |
|         poolformer_m36          | 1  |  0.7614  |
|           convit_base           | 1  |  0.7574  |
|      vit_base_patch16_224       | 1  |  0.755   |
|          cait_m36_384           | 1  |  0.7391  |
|           tf_mixnet_l           | 1  |  0.7347  |
|          mixer_b16_224          | 1  |  0.7273  |
|             dla102              | 1  |  0.7187  |
|            mixnet_l             | 1  |  0.7127  |
|          resmlp_12_224          | 1  |  0.6882  |
|            pit_b_224            | 1  |  0.6741  |
|        twins_pcpvt_base         | 1  |  0.6635  |
|        tnt_s_patch16_224        | 1  |  0.6476  |
|          jx_nest_base           | 1  |  0.6457  |
|         coat_lite_mini          | 1  |  0.6293  |
|           rexnet_100            | 1  |  0.6235  |
|         crossvit_9_240          | 1  |  0.6016  |
|           mobilevit_s           | 1  |  0.5561  |
|            tinynet_a            | 1  |   0.5    |
|       tf_efficientnet_b0        | 1  |  0.4917  |
|          gmlp_s16_224           | 1  |  0.4346  |
|     swsl_resnext101_32x16d      | 1  |  0.0673  |
|       eca_botnext26ts_256       | 0  |   0.0    |
|        sebotnet33ts_256         | 0  |   0.0    |
|          botnet26t_256          | 0  |   0.0    |
|        eca_halonext26ts         | 0  |   0.0    |
|        convmixer_768_32         | 0  |   0.0    |
+---------------------------------+----+----------+

Accuracy

+---------------------------------+----+---------------+
|              name               | bs |   inductor    |
+---------------------------------+----+---------------+
|        adv_inception_v3         | 1  |     pass      |
|             dpn107              | 1  |     pass      |
|      beit_base_patch16_224      | 1  |     pass      |
|          mixer_b16_224          | 1  |     pass      |
|        ese_vovnet19b_dw         | 1  |     pass      |
|         coat_lite_mini          | 1  |     pass      |
|           convit_base           | 1  |     pass      |
|          convnext_base          | 1  |     pass      |
|         crossvit_9_240          | 1  |     pass      |
|          cspdarknet53           | 1  |     pass      |
| deit_base_distilled_patch16_224 | 1  |     pass      |
|             dla102              | 1  |     pass      |
|           dm_nfnet_f0           | 1  |     pass      |
|            lcnet_050            | 1  |     pass      |
|           volo_d1_224           | 1  |     pass      |
|      xcit_large_24_p8_224       | 1  |     pass      |
|           fbnetc_100            | 1  |     pass      |
|        gluon_xception65         | 1  |     pass      |
|          jx_nest_base           | 1  |     pass      |
|          inception_v3           | 1  |     pass      |
|            hrnet_w18            | 1  |     pass      |
|          gmlp_s16_224           | 1  |     pass      |
|          gmixer_24_224          | 1  |     pass      |
|       gluon_inception_v3        | 1  |     pass      |
|            gernet_l             | 1  |     pass      |
|            fbnetv3_b            | 1  |     pass      |
|           mnasnet_100           | 1  |     pass      |
|            mixnet_l             | 1  |     pass      |
|      vit_base_patch16_224       | 1  |     pass      |
|           res2next50            | 1  |     pass      |
|         mobilenetv2_100         | 1  |     pass      |
|      mobilenetv3_large_100      | 1  |     pass      |
|           mobilevit_s           | 1  |     pass      |
|            nfnet_l0             | 1  |     pass      |
|            pit_b_224            | 1  |     pass      |
|          pnasnet5large          | 1  |     pass      |
|         poolformer_m36          | 1  |     pass      |
|           regnety_002           | 1  |     pass      |
|            repvgg_a2            | 1  |     pass      |
|         visformer_small         | 1  |     pass      |
|        res2net50_14w_8s         | 1  |     pass      |
|        res2net101_26w_4s        | 1  |     pass      |
|          resmlp_12_224          | 1  |     pass      |
|       tf_efficientnet_b0        | 1  |     pass      |
|        twins_pcpvt_base         | 1  |     pass      |
|        tnt_s_patch16_224        | 1  |     pass      |
|           resnest101e           | 1  |     pass      |
|           tf_mixnet_l           | 1  |     pass      |
|            tinynet_a            | 1  |     pass      |
|     swsl_resnext101_32x16d      | 1  |     pass      |
|  swin_base_patch4_window7_224   | 1  |     pass      |
|          spnasnet_100           | 1  |     pass      |
|           selecsls42b           | 1  |     pass      |
|           rexnet_100            | 1  |     pass      |
|        eca_halonext26ts         | 1  |  fail_to_run  |
|        convmixer_768_32         | 1  |  fail_to_run  |
|        sebotnet33ts_256         | 1  |  fail_to_run  |
|          botnet26t_256          | 1  |  fail_to_run  |
|       eca_botnext26ts_256       | 1  |  fail_to_run  |
|          ghostnet_100           | 1  | fail_accuracy |
|          cait_m36_384           | 1  | fail_accuracy |
+---------------------------------+----+---------------+

Compilation latency (sec)

+---------------------------------+----+----------+
|              name               | bs | inductor |
+---------------------------------+----+----------+
|     swsl_resnext101_32x16d      | 1  | 87.5279  |
|          pnasnet5large          | 1  | 46.3015  |
|          cait_m36_384           | 1  | 36.1308  |
|           tf_mixnet_l           | 1  |  35.475  |
|            hrnet_w18            | 1  | 33.5026  |
|  swin_base_patch4_window7_224   | 1  | 33.1088  |
|        twins_pcpvt_base         | 1  | 33.0708  |
|           rexnet_100            | 1  | 32.9664  |
|           mobilevit_s           | 1  | 32.2259  |
|            mixnet_l             | 1  | 31.5505  |
|      xcit_large_24_p8_224       | 1  |  30.044  |
|           resnest101e           | 1  | 29.2334  |
|            fbnetv3_b            | 1  | 29.1483  |
|        adv_inception_v3         | 1  | 29.0574  |
|             dpn107              | 1  | 28.7993  |
|        res2net50_14w_8s         | 1  | 28.4829  |
|           dm_nfnet_f0           | 1  | 27.8804  |
|          ghostnet_100           | 1  |  27.737  |
|         poolformer_m36          | 1  | 26.9426  |
|       tf_efficientnet_b0        | 1  | 26.5395  |
|            tinynet_a            | 1  | 26.4855  |
|        res2net101_26w_4s        | 1  | 25.9801  |
|          jx_nest_base           | 1  | 24.9393  |
|         coat_lite_mini          | 1  | 24.2566  |
|            nfnet_l0             | 1  | 24.0727  |
|      mobilenetv3_large_100      | 1  |  23.634  |
|        tnt_s_patch16_224        | 1  | 23.2213  |
|           fbnetc_100            | 1  | 23.1681  |
|             dla102              | 1  | 23.0499  |
|           volo_d1_224           | 1  | 22.8963  |
|         crossvit_9_240          | 1  |  22.406  |
|          convnext_base          | 1  | 21.0074  |
|          cspdarknet53           | 1  | 20.9592  |
|           res2next50            | 1  | 20.4985  |
|       gluon_inception_v3        | 1  | 20.1694  |
|            pit_b_224            | 1  | 20.0287  |
|          spnasnet_100           | 1  | 19.9673  |
|          inception_v3           | 1  | 19.8218  |
|          gmlp_s16_224           | 1  | 19.3607  |
|           regnety_002           | 1  | 18.9754  |
|         mobilenetv2_100         | 1  | 18.4704  |
|        gluon_xception65         | 1  | 18.4543  |
|           mnasnet_100           | 1  | 18.4216  |
|           convit_base           | 1  | 18.2287  |
|         visformer_small         | 1  | 17.5961  |
|            gernet_l             | 1  | 16.5629  |
|          gmixer_24_224          | 1  | 16.3941  |
|           selecsls42b           | 1  | 16.1093  |
|        ese_vovnet19b_dw         | 1  | 15.1091  |
|          mixer_b16_224          | 1  | 15.0007  |
|            repvgg_a2            | 1  | 14.8364  |
| deit_base_distilled_patch16_224 | 1  | 14.7708  |
|      vit_base_patch16_224       | 1  | 14.6892  |
|      beit_base_patch16_224      | 1  | 14.5156  |
|            lcnet_050            | 1  | 14.3073  |
|          resmlp_12_224          | 1  | 11.9188  |
|          botnet26t_256          | 0  |   nan    |
|        convmixer_768_32         | 0  |   nan    |
|       eca_botnext26ts_256       | 0  |   nan    |
|        eca_halonext26ts         | 0  |   nan    |
|        sebotnet33ts_256         | 0  |   nan    |
+---------------------------------+----+----------+

Peak Memory Compression Ratio

+---------------------------------+----+----------+
|              name               | bs | inductor |
+---------------------------------+----+----------+
|      xcit_large_24_p8_224       | 1  |  2.2401  |
|          cait_m36_384           | 1  |  2.165   |
|          pnasnet5large          | 1  |  1.3583  |
|           dm_nfnet_f0           | 1  |  1.2227  |
|            nfnet_l0             | 1  |  1.2121  |
|           mobilevit_s           | 1  |  1.2001  |
|          jx_nest_base           | 1  |  1.1911  |
|             dpn107              | 1  |  1.1863  |
|          convnext_base          | 1  |  1.1528  |
|           convit_base           | 1  |  1.1385  |
|          cspdarknet53           | 1  |  1.1357  |
|         poolformer_m36          | 1  |  1.1085  |
|          gmixer_24_224          | 1  |  1.1051  |
|  swin_base_patch4_window7_224   | 1  |  1.1047  |
|      beit_base_patch16_224      | 1  |  1.0966  |
|           volo_d1_224           | 1  |  1.0917  |
| deit_base_distilled_patch16_224 | 1  |  1.0801  |
|      vit_base_patch16_224       | 1  |   1.08   |
|        tnt_s_patch16_224        | 1  |  1.0799  |
|          mixer_b16_224          | 1  |  1.0769  |
|         mobilenetv2_100         | 1  |  1.0659  |
|        twins_pcpvt_base         | 1  |  1.0643  |
|           tf_mixnet_l           | 1  |  1.0592  |
|        ese_vovnet19b_dw         | 1  |  1.0515  |
|       tf_efficientnet_b0        | 1  |  1.0506  |
|           rexnet_100            | 1  |  1.0491  |
|         coat_lite_mini          | 1  |  1.0484  |
|          resmlp_12_224          | 1  |  1.0453  |
|            mixnet_l             | 1  |  1.0396  |
|            fbnetv3_b            | 1  |  1.0266  |
|            gernet_l             | 1  |  1.0235  |
|            tinynet_a            | 1  |  1.0175  |
|          spnasnet_100           | 1  |  1.0163  |
|           fbnetc_100            | 1  |  1.011   |
|           mnasnet_100           | 1  |  1.0069  |
|            repvgg_a2            | 1  |  1.0065  |
|           resnest101e           | 1  |  0.9914  |
|            pit_b_224            | 1  |  0.9784  |
|      mobilenetv3_large_100      | 1  |  0.9722  |
|            lcnet_050            | 1  |  0.9651  |
|            hrnet_w18            | 1  |  0.9589  |
|           res2next50            | 1  |  0.9575  |
|           regnety_002           | 1  |  0.9549  |
|         visformer_small         | 1  |  0.9525  |
|       gluon_inception_v3        | 1  |  0.9516  |
|        adv_inception_v3         | 1  |  0.9507  |
|          inception_v3           | 1  |  0.9499  |
|             dla102              | 1  |  0.9464  |
|          ghostnet_100           | 1  |  0.9441  |
|        res2net50_14w_8s         | 1  |  0.941   |
|         crossvit_9_240          | 1  |  0.8815  |
|          gmlp_s16_224           | 1  |   0.88   |
|     swsl_resnext101_32x16d      | 1  |  0.8433  |
|           selecsls42b           | 1  |  0.8197  |
|        gluon_xception65         | 1  |  0.8117  |
|        res2net101_26w_4s        | 1  |  0.7678  |
|          botnet26t_256          | 0  |   nan    |
|        convmixer_768_32         | 0  |   nan    |
|       eca_botnext26ts_256       | 0  |   nan    |
|        eca_halonext26ts         | 0  |   nan    |
|        sebotnet33ts_256         | 0  |   nan    |
+---------------------------------+----+----------+