[Inductor] incorrect result of vision_maskrcnn

[blzheng]

🐛 Describe the bug

Vision_maskrcnn of torchbench has incorrect result when using inductor.
RuntimeError: The size of tensor a (36) must match the size of tensor b (34) at non-singleton dimension 0

Reproduce command:
python benchmarks/dynamo/torchbench.py --accuracy --float32 -dcpu --inductor-settings --float32 -n50 --inductor --no-skip --quiet -k "vision_maskrcnn"

Error logs

cpu eval vision_maskrcnn
Traceback (most recent call last):
File "benchmarks/dynamo/torchbench.py", line 349, in
main(TorchBenchmarkRunner(), original_dir)
File "/home/bzheng/workspace/pytorch/benchmarks/dynamo/common.py", line 1532, in main
return maybe_fresh_cache(run, args.cold_start_latency and args.only)(
File "/home/bzheng/workspace/pytorch/benchmarks/dynamo/common.py", line 772, in inner
return fn(*args, **kwargs)
File "/home/bzheng/workspace/pytorch/benchmarks/dynamo/common.py", line 1845, in run
runner.run_one_model(
File "/home/bzheng/workspace/pytorch/benchmarks/dynamo/common.py", line 1267, in run_one_model
status = self.check_accuracy(
File "/home/bzheng/workspace/pytorch/benchmarks/dynamo/common.py", line 1104, in check_accuracy
if not same(
File "/home/bzheng/workspace/pytorch/torch/_dynamo/utils.py", line 774, in same
return len(ref) == len(res) and all(
File "/home/bzheng/workspace/pytorch/torch/_dynamo/utils.py", line 775, in
same(ai, bi, fp64_refi, cos_similarity, tol, equal_nan, exact_dtype)
File "/home/bzheng/workspace/pytorch/torch/_dynamo/utils.py", line 785, in same
same(
File "/home/bzheng/workspace/pytorch/torch/_dynamo/utils.py", line 831, in same
if torch.allclose(ref, res, atol=tol, rtol=tol, equal_nan=equal_nan):
RuntimeError: The size of tensor a (36) must match the size of tensor b (34) at non-singleton dimension 0
ERROR
cpu gmean=nanx mean=nanx
vision_maskrcnn gmean=nanx mean=nanx
0 gmean=nanx mean=nanx
0.0000 gmean=nanx mean=nanx

Minified repro

python benchmarks/dynamo/torchbench.py --accuracy --float32 -dcpu --inductor-settings --float32 -n50 --inductor --no-skip --quiet -k "vision_maskrcnn"

[Valentine233]

I will take a look on this issue.

[Valentine233]

Update 11/04: The issue is about the output size mismatch. There are two input examples in UT, respectively with sizes (3, 612, 612) and (3, 427, 640). The first one with equal height and width can pass the UT, while the second one reports mismatched output size. I have narrowed down the problem to the two successive FX graphs.

FX graph 34

class <lambda>(torch.nn.Module):
    def forward(self, arg0_1: f32[1000, 91], arg1_1: f32[1000, 364], arg2_1: f32[1000, 4]):
        # File: /home/neo/anaconda3/lib/python3.9/site-packages/torchvision-0.15.0a0+d95fbaf-py3.9-linux-x86_64.egg/torchvision/models/detection/_utils.py:172, code: concat_boxes = torch.cat(boxes, dim=0)
        cat: f32[1000, 4] = torch.ops.aten.cat.default([arg2_1]);  arg2_1 = None
        
        # File: /home/neo/anaconda3/lib/python3.9/site-packages/torchvision-0.15.0a0+d95fbaf-py3.9-linux-x86_64.egg/torchvision/models/detection/_utils.py:177, code: rel_codes = rel_codes.reshape(box_sum, -1)
        view: f32[1000, 364] = torch.ops.aten.view.default(arg1_1, [1000, 364]);  arg1_1 = None
        
        # File: /home/neo/anaconda3/lib/python3.9/site-packages/torchvision-0.15.0a0+d95fbaf-py3.9-linux-x86_64.egg/torchvision/models/detection/_utils.py:195, code: widths = boxes[:, 2] - boxes[:, 0]
        slice_1: f32[1000, 4] = torch.ops.aten.slice.Tensor(cat, 0, 0, 9223372036854775807)
        select: f32[1000] = torch.ops.aten.select.int(slice_1, 1, 2);  slice_1 = None
        slice_2: f32[1000, 4] = torch.ops.aten.slice.Tensor(cat, 0, 0, 9223372036854775807)
        select_1: f32[1000] = torch.ops.aten.select.int(slice_2, 1, 0);  slice_2 = None
        sub: f32[1000] = torch.ops.aten.sub.Tensor(select, select_1);  select = select_1 = None
        
        # File: /home/neo/anaconda3/lib/python3.9/site-packages/torchvision-0.15.0a0+d95fbaf-py3.9-linux-x86_64.egg/torchvision/models/detection/_utils.py:196, code: heights = boxes[:, 3] - boxes[:, 1]
        slice_3: f32[1000, 4] = torch.ops.aten.slice.Tensor(cat, 0, 0, 9223372036854775807)
        select_2: f32[1000] = torch.ops.aten.select.int(slice_3, 1, 3);  slice_3 = None
        slice_4: f32[1000, 4] = torch.ops.aten.slice.Tensor(cat, 0, 0, 9223372036854775807)
        select_3: f32[1000] = torch.ops.aten.select.int(slice_4, 1, 1);  slice_4 = None
        sub_1: f32[1000] = torch.ops.aten.sub.Tensor(select_2, select_3);  select_2 = select_3 = None
        
        # File: /home/neo/anaconda3/lib/python3.9/site-packages/torchvision-0.15.0a0+d95fbaf-py3.9-linux-x86_64.egg/torchvision/models/detection/_utils.py:197, code: ctr_x = boxes[:, 0] + 0.5 * widths
        slice_5: f32[1000, 4] = torch.ops.aten.slice.Tensor(cat, 0, 0, 9223372036854775807)
        select_4: f32[1000] = torch.ops.aten.select.int(slice_5, 1, 0);  slice_5 = None
        mul: f32[1000] = torch.ops.aten.mul.Tensor(sub, 0.5)
        add: f32[1000] = torch.ops.aten.add.Tensor(select_4, mul);  select_4 = mul = None
        
        # File: /home/neo/anaconda3/lib/python3.9/site-packages/torchvision-0.15.0a0+d95fbaf-py3.9-linux-x86_64.egg/torchvision/models/detection/_utils.py:198, code: ctr_y = boxes[:, 1] + 0.5 * heights
        slice_6: f32[1000, 4] = torch.ops.aten.slice.Tensor(cat, 0, 0, 9223372036854775807);  cat = None
        select_5: f32[1000] = torch.ops.aten.select.int(slice_6, 1, 1);  slice_6 = None
        mul_1: f32[1000] = torch.ops.aten.mul.Tensor(sub_1, 0.5)
        add_1: f32[1000] = torch.ops.aten.add.Tensor(select_5, mul_1);  select_5 = mul_1 = None
        
        # File: /home/neo/anaconda3/lib/python3.9/site-packages/torchvision-0.15.0a0+d95fbaf-py3.9-linux-x86_64.egg/torchvision/models/detection/_utils.py:201, code: dx = rel_codes[:, 0::4] / wx
        slice_7: f32[1000, 364] = torch.ops.aten.slice.Tensor(view, 0, 0, 9223372036854775807)
        slice_8: f32[1000, 91] = torch.ops.aten.slice.Tensor(slice_7, 1, 0, 9223372036854775807, 4);  slice_7 = None
        div: f32[1000, 91] = torch.ops.aten.div.Tensor(slice_8, 10.0);  slice_8 = None
        
        # File: /home/neo/anaconda3/lib/python3.9/site-packages/torchvision-0.15.0a0+d95fbaf-py3.9-linux-x86_64.egg/torchvision/models/detection/_utils.py:202, code: dy = rel_codes[:, 1::4] / wy
        slice_9: f32[1000, 364] = torch.ops.aten.slice.Tensor(view, 0, 0, 9223372036854775807)
        slice_10: f32[1000, 91] = torch.ops.aten.slice.Tensor(slice_9, 1, 1, 9223372036854775807, 4);  slice_9 = None
        div_1: f32[1000, 91] = torch.ops.aten.div.Tensor(slice_10, 10.0);  slice_10 = None
        
        # File: /home/neo/anaconda3/lib/python3.9/site-packages/torchvision-0.15.0a0+d95fbaf-py3.9-linux-x86_64.egg/torchvision/models/detection/_utils.py:203, code: dw = rel_codes[:, 2::4] / ww
        slice_11: f32[1000, 364] = torch.ops.aten.slice.Tensor(view, 0, 0, 9223372036854775807)
        slice_12: f32[1000, 91] = torch.ops.aten.slice.Tensor(slice_11, 1, 2, 9223372036854775807, 4);  slice_11 = None
        div_2: f32[1000, 91] = torch.ops.aten.div.Tensor(slice_12, 5.0);  slice_12 = None
        
        # File: /home/neo/anaconda3/lib/python3.9/site-packages/torchvision-0.15.0a0+d95fbaf-py3.9-linux-x86_64.egg/torchvision/models/detection/_utils.py:204, code: dh = rel_codes[:, 3::4] / wh
        slice_13: f32[1000, 364] = torch.ops.aten.slice.Tensor(view, 0, 0, 9223372036854775807);  view = None
        slice_14: f32[1000, 91] = torch.ops.aten.slice.Tensor(slice_13, 1, 3, 9223372036854775807, 4);  slice_13 = None
        div_3: f32[1000, 91] = torch.ops.aten.div.Tensor(slice_14, 5.0);  slice_14 = None
        
        # No stacktrace found for following nodes
        _tensor_constant0 = self._tensor_constant0
        
        # File: /home/neo/anaconda3/lib/python3.9/site-packages/torchvision-0.15.0a0+d95fbaf-py3.9-linux-x86_64.egg/torchvision/models/detection/_utils.py:207, code: dw = torch.clamp(dw, max=self.bbox_xform_clip)
        lift_fresh_copy: f32[] = torch.ops.aten.lift_fresh_copy.default(_tensor_constant0);  _tensor_constant0 = None
        minimum: f32[1000, 91] = torch.ops.aten.minimum.default(div_2, lift_fresh_copy);  div_2 = lift_fresh_copy = None
        
        # No stacktrace found for following nodes
        _tensor_constant1 = self._tensor_constant1
        
        # File: /home/neo/anaconda3/lib/python3.9/site-packages/torchvision-0.15.0a0+d95fbaf-py3.9-linux-x86_64.egg/torchvision/models/detection/_utils.py:208, code: dh = torch.clamp(dh, max=self.bbox_xform_clip)
        lift_fresh_copy_1: f32[] = torch.ops.aten.lift_fresh_copy.default(_tensor_constant1);  _tensor_constant1 = None
        minimum_1: f32[1000, 91] = torch.ops.aten.minimum.default(div_3, lift_fresh_copy_1);  div_3 = lift_fresh_copy_1 = None
        
        # File: /home/neo/anaconda3/lib/python3.9/site-packages/torchvision-0.15.0a0+d95fbaf-py3.9-linux-x86_64.egg/torchvision/models/detection/_utils.py:210, code: pred_ctr_x = dx * widths[:, None] + ctr_x[:, None]
        slice_15: f32[1000] = torch.ops.aten.slice.Tensor(sub, 0, 0, 9223372036854775807)
        unsqueeze: f32[1000, 1] = torch.ops.aten.unsqueeze.default(slice_15, 1);  slice_15 = None
        mul_2: f32[1000, 91] = torch.ops.aten.mul.Tensor(div, unsqueeze);  div = unsqueeze = None
        slice_16: f32[1000] = torch.ops.aten.slice.Tensor(add, 0, 0, 9223372036854775807);  add = None
        unsqueeze_1: f32[1000, 1] = torch.ops.aten.unsqueeze.default(slice_16, 1);  slice_16 = None
        add_2: f32[1000, 91] = torch.ops.aten.add.Tensor(mul_2, unsqueeze_1);  mul_2 = unsqueeze_1 = None
        
        # File: /home/neo/anaconda3/lib/python3.9/site-packages/torchvision-0.15.0a0+d95fbaf-py3.9-linux-x86_64.egg/torchvision/models/detection/_utils.py:211, code: pred_ctr_y = dy * heights[:, None] + ctr_y[:, None]
        slice_17: f32[1000] = torch.ops.aten.slice.Tensor(sub_1, 0, 0, 9223372036854775807)
        unsqueeze_2: f32[1000, 1] = torch.ops.aten.unsqueeze.default(slice_17, 1);  slice_17 = None
        mul_3: f32[1000, 91] = torch.ops.aten.mul.Tensor(div_1, unsqueeze_2);  div_1 = unsqueeze_2 = None
        slice_18: f32[1000] = torch.ops.aten.slice.Tensor(add_1, 0, 0, 9223372036854775807);  add_1 = None
        unsqueeze_3: f32[1000, 1] = torch.ops.aten.unsqueeze.default(slice_18, 1);  slice_18 = None
        add_3: f32[1000, 91] = torch.ops.aten.add.Tensor(mul_3, unsqueeze_3);  mul_3 = unsqueeze_3 = None
        
        # File: /home/neo/anaconda3/lib/python3.9/site-packages/torchvision-0.15.0a0+d95fbaf-py3.9-linux-x86_64.egg/torchvision/models/detection/_utils.py:212, code: pred_w = torch.exp(dw) * widths[:, None]
        exp: f32[1000, 91] = torch.ops.aten.exp.default(minimum);  minimum = None
        slice_19: f32[1000] = torch.ops.aten.slice.Tensor(sub, 0, 0, 9223372036854775807);  sub = None
        unsqueeze_4: f32[1000, 1] = torch.ops.aten.unsqueeze.default(slice_19, 1);  slice_19 = None
        mul_4: f32[1000, 91] = torch.ops.aten.mul.Tensor(exp, unsqueeze_4);  exp = unsqueeze_4 = None
        
        # File: /home/neo/anaconda3/lib/python3.9/site-packages/torchvision-0.15.0a0+d95fbaf-py3.9-linux-x86_64.egg/torchvision/models/detection/_utils.py:213, code: pred_h = torch.exp(dh) * heights[:, None]
        exp_1: f32[1000, 91] = torch.ops.aten.exp.default(minimum_1);  minimum_1 = None
        slice_20: f32[1000] = torch.ops.aten.slice.Tensor(sub_1, 0, 0, 9223372036854775807);  sub_1 = None
        unsqueeze_5: f32[1000, 1] = torch.ops.aten.unsqueeze.default(slice_20, 1);  slice_20 = None
        mul_5: f32[1000, 91] = torch.ops.aten.mul.Tensor(exp_1, unsqueeze_5);  exp_1 = unsqueeze_5 = None
        
        # No stacktrace found for following nodes
        _tensor_constant2 = self._tensor_constant2
        
        # File: /home/neo/anaconda3/lib/python3.9/site-packages/torchvision-0.15.0a0+d95fbaf-py3.9-linux-x86_64.egg/torchvision/models/detection/_utils.py:216, code: c_to_c_h = torch.tensor(0.5, dtype=pred_ctr_y.dtype, device=pred_h.device) * pred_h
        lift_fresh_copy_2: f32[] = torch.ops.aten.lift_fresh_copy.default(_tensor_constant2);  _tensor_constant2 = None
        mul_6: f32[1000, 91] = torch.ops.aten.mul.Tensor(lift_fresh_copy_2, mul_5);  lift_fresh_copy_2 = mul_5 = None
        
        # No stacktrace found for following nodes
        _tensor_constant3 = self._tensor_constant3
        
        # File: /home/neo/anaconda3/lib/python3.9/site-packages/torchvision-0.15.0a0+d95fbaf-py3.9-linux-x86_64.egg/torchvision/models/detection/_utils.py:217, code: c_to_c_w = torch.tensor(0.5, dtype=pred_ctr_x.dtype, device=pred_w.device) * pred_w
        lift_fresh_copy_3: f32[] = torch.ops.aten.lift_fresh_copy.default(_tensor_constant3);  _tensor_constant3 = None
        mul_7: f32[1000, 91] = torch.ops.aten.mul.Tensor(lift_fresh_copy_3, mul_4);  lift_fresh_copy_3 = mul_4 = None
        
        # File: /home/neo/anaconda3/lib/python3.9/site-packages/torchvision-0.15.0a0+d95fbaf-py3.9-linux-x86_64.egg/torchvision/models/detection/_utils.py:219, code: pred_boxes1 = pred_ctr_x - c_to_c_w
        sub_2: f32[1000, 91] = torch.ops.aten.sub.Tensor(add_2, mul_7)
        
        # File: /home/neo/anaconda3/lib/python3.9/site-packages/torchvision-0.15.0a0+d95fbaf-py3.9-linux-x86_64.egg/torchvision/models/detection/_utils.py:220, code: pred_boxes2 = pred_ctr_y - c_to_c_h
        sub_3: f32[1000, 91] = torch.ops.aten.sub.Tensor(add_3, mul_6)
        
        # File: /home/neo/anaconda3/lib/python3.9/site-packages/torchvision-0.15.0a0+d95fbaf-py3.9-linux-x86_64.egg/torchvision/models/detection/_utils.py:221, code: pred_boxes3 = pred_ctr_x + c_to_c_w
        add_4: f32[1000, 91] = torch.ops.aten.add.Tensor(add_2, mul_7);  add_2 = mul_7 = None
        
        # File: /home/neo/anaconda3/lib/python3.9/site-packages/torchvision-0.15.0a0+d95fbaf-py3.9-linux-x86_64.egg/torchvision/models/detection/_utils.py:222, code: pred_boxes4 = pred_ctr_y + c_to_c_h
        add_5: f32[1000, 91] = torch.ops.aten.add.Tensor(add_3, mul_6);  add_3 = mul_6 = None
        
        # File: /home/neo/anaconda3/lib/python3.9/site-packages/torchvision-0.15.0a0+d95fbaf-py3.9-linux-x86_64.egg/torchvision/models/detection/_utils.py:223, code: pred_boxes = torch.stack((pred_boxes1, pred_boxes2, pred_boxes3, pred_boxes4), dim=2).flatten(1)
        unsqueeze_6: f32[1000, 91, 1] = torch.ops.aten.unsqueeze.default(sub_2, 2);  sub_2 = None
        unsqueeze_7: f32[1000, 91, 1] = torch.ops.aten.unsqueeze.default(sub_3, 2);  sub_3 = None
        unsqueeze_8: f32[1000, 91, 1] = torch.ops.aten.unsqueeze.default(add_4, 2);  add_4 = None
        unsqueeze_9: f32[1000, 91, 1] = torch.ops.aten.unsqueeze.default(add_5, 2);  add_5 = None
        cat_1: f32[1000, 91, 4] = torch.ops.aten.cat.default([unsqueeze_6, unsqueeze_7, unsqueeze_8, unsqueeze_9], 2);  unsqueeze_6 = unsqueeze_7 = unsqueeze_8 = unsqueeze_9 = None
        view_1: f32[1000, 364] = torch.ops.aten.view.default(cat_1, [1000, 364]);  cat_1 = None
        
        # File: /home/neo/anaconda3/lib/python3.9/site-packages/torchvision-0.15.0a0+d95fbaf-py3.9-linux-x86_64.egg/torchvision/models/detection/_utils.py:180, code: pred_boxes = pred_boxes.reshape(box_sum, -1, 4)
        view_2: f32[1000, 91, 4] = torch.ops.aten.view.default(view_1, [1000, 91, 4]);  view_1 = None
        
        # File: /home/neo/anaconda3/lib/python3.9/site-packages/torchvision-0.15.0a0+d95fbaf-py3.9-linux-x86_64.egg/torchvision/models/detection/roi_heads.py:682, code: pred_scores = F.softmax(class_logits, -1)
        amax: f32[1000, 1] = torch.ops.aten.amax.default(arg0_1, [-1], True)
        sub_4: f32[1000, 91] = torch.ops.aten.sub.Tensor(arg0_1, amax);  arg0_1 = amax = None
        exp_2: f32[1000, 91] = torch.ops.aten.exp.default(sub_4);  sub_4 = None
        sum_1: f32[1000, 1] = torch.ops.aten.sum.dim_IntList(exp_2, [-1], True)
        div_4: f32[1000, 91] = torch.ops.aten.div.Tensor(exp_2, sum_1);  exp_2 = sum_1 = None
        
        # File: /home/neo/anaconda3/lib/python3.9/site-packages/torchvision-0.15.0a0+d95fbaf-py3.9-linux-x86_64.egg/torchvision/models/detection/roi_heads.py:684, code: pred_boxes_list = pred_boxes.split(boxes_per_image, 0)
        split_with_sizes = torch.ops.aten.split_with_sizes.default(view_2, [1000]);  view_2 = None
        getitem: f32[1000, 91, 4] = split_with_sizes[0];  split_with_sizes = None
        
        # File: /home/neo/anaconda3/lib/python3.9/site-packages/torchvision-0.15.0a0+d95fbaf-py3.9-linux-x86_64.egg/torchvision/models/detection/roi_heads.py:685, code: pred_scores_list = pred_scores.split(boxes_per_image, 0)
        split_with_sizes_1 = torch.ops.aten.split_with_sizes.default(div_4, [1000]);  div_4 = None
        getitem_1: f32[1000, 91] = split_with_sizes_1[0];  split_with_sizes_1 = None
        
        # File: /home/neo/anaconda3/lib/python3.9/site-packages/torchvision-0.15.0a0+d95fbaf-py3.9-linux-x86_64.egg/torchvision/ops/boxes.py:151, code: boxes_x = boxes[..., 0::2]
        slice_21: f32[1000, 91, 2] = torch.ops.aten.slice.Tensor(getitem, 2, 0, 9223372036854775807, 2)
        
        # File: /home/neo/anaconda3/lib/python3.9/site-packages/torchvision-0.15.0a0+d95fbaf-py3.9-linux-x86_64.egg/torchvision/ops/boxes.py:152, code: boxes_y = boxes[..., 1::2]
        slice_22: f32[1000, 91, 2] = torch.ops.aten.slice.Tensor(getitem, 2, 1, 9223372036854775807, 2);  getitem = None
        
        # No stacktrace found for following nodes
        _tensor_constant4 = self._tensor_constant4
        
        # File: /home/neo/anaconda3/lib/python3.9/site-packages/torchvision-0.15.0a0+d95fbaf-py3.9-linux-x86_64.egg/torchvision/ops/boxes.py:161, code: boxes_x = boxes_x.clamp(min=0, max=width)
        lift_fresh_copy_4: f32[] = torch.ops.aten.lift_fresh_copy.default(_tensor_constant4);  _tensor_constant4 = None
        maximum: f32[1000, 91, 2] = torch.ops.aten.maximum.default(slice_21, lift_fresh_copy_4);  slice_21 = lift_fresh_copy_4 = None
        
        # No stacktrace found for following nodes
        _tensor_constant5 = self._tensor_constant5
        
        # File: /home/neo/anaconda3/lib/python3.9/site-packages/torchvision-0.15.0a0+d95fbaf-py3.9-linux-x86_64.egg/torchvision/ops/boxes.py:161, code: boxes_x = boxes_x.clamp(min=0, max=width)
        lift_fresh_copy_5: f32[] = torch.ops.aten.lift_fresh_copy.default(_tensor_constant5);  _tensor_constant5 = None
        minimum_2: f32[1000, 91, 2] = torch.ops.aten.minimum.default(maximum, lift_fresh_copy_5);  maximum = lift_fresh_copy_5 = None
        
        # No stacktrace found for following nodes
        _tensor_constant6 = self._tensor_constant6
        
        # File: /home/neo/anaconda3/lib/python3.9/site-packages/torchvision-0.15.0a0+d95fbaf-py3.9-linux-x86_64.egg/torchvision/ops/boxes.py:162, code: boxes_y = boxes_y.clamp(min=0, max=height)
        lift_fresh_copy_6: f32[] = torch.ops.aten.lift_fresh_copy.default(_tensor_constant6);  _tensor_constant6 = None
        maximum_1: f32[1000, 91, 2] = torch.ops.aten.maximum.default(slice_22, lift_fresh_copy_6);  slice_22 = lift_fresh_copy_6 = None
        
        # No stacktrace found for following nodes
        _tensor_constant7 = self._tensor_constant7
        
        # File: /home/neo/anaconda3/lib/python3.9/site-packages/torchvision-0.15.0a0+d95fbaf-py3.9-linux-x86_64.egg/torchvision/ops/boxes.py:162, code: boxes_y = boxes_y.clamp(min=0, max=height)
        lift_fresh_copy_7: f32[] = torch.ops.aten.lift_fresh_copy.default(_tensor_constant7);  _tensor_constant7 = None
        minimum_3: f32[1000, 91, 2] = torch.ops.aten.minimum.default(maximum_1, lift_fresh_copy_7);  maximum_1 = lift_fresh_copy_7 = None
        
        # File: /home/neo/anaconda3/lib/python3.9/site-packages/torchvision-0.15.0a0+d95fbaf-py3.9-linux-x86_64.egg/torchvision/ops/boxes.py:164, code: clipped_boxes = torch.stack((boxes_x, boxes_y), dim=dim)
        unsqueeze_10: f32[1000, 91, 2, 1] = torch.ops.aten.unsqueeze.default(minimum_2, 3);  minimum_2 = None
        unsqueeze_11: f32[1000, 91, 2, 1] = torch.ops.aten.unsqueeze.default(minimum_3, 3);  minimum_3 = None
        cat_2: f32[1000, 91, 2, 2] = torch.ops.aten.cat.default([unsqueeze_10, unsqueeze_11], 3);  unsqueeze_10 = unsqueeze_11 = None
        
        # File: /home/neo/anaconda3/lib/python3.9/site-packages/torchvision-0.15.0a0+d95fbaf-py3.9-linux-x86_64.egg/torchvision/ops/boxes.py:165, code: return clipped_boxes.reshape(boxes.shape)
        view_3: f32[1000, 91, 4] = torch.ops.aten.view.default(cat_2, [1000, 91, 4]);  cat_2 = None
        
        # File: /home/neo/anaconda3/lib/python3.9/site-packages/torchvision-0.15.0a0+d95fbaf-py3.9-linux-x86_64.egg/torchvision/models/detection/roi_heads.py:694, code: labels = torch.arange(num_classes, device=device)
        arange: i64[91] = torch.ops.aten.arange.default(91, device = device(type='cpu'), pin_memory = False)
        alias: i64[91] = torch.ops.aten.alias.default(arange);  arange = None
        
        # File: /home/neo/anaconda3/lib/python3.9/site-packages/torchvision-0.15.0a0+d95fbaf-py3.9-linux-x86_64.egg/torchvision/models/detection/roi_heads.py:695, code: labels = labels.view(1, -1).expand_as(scores)
        view_4: i64[1, 91] = torch.ops.aten.view.default(alias, [1, -1]);  alias = None
        expand: i64[1000, 91] = torch.ops.aten.expand.default(view_4, [1000, 91]);  view_4 = None
        
        # File: /home/neo/anaconda3/lib/python3.9/site-packages/torchvision-0.15.0a0+d95fbaf-py3.9-linux-x86_64.egg/torchvision/models/detection/roi_heads.py:698, code: boxes = boxes[:, 1:]
        slice_23: f32[1000, 91, 4] = torch.ops.aten.slice.Tensor(view_3, 0, 0, 9223372036854775807);  view_3 = None
        slice_24: f32[1000, 90, 4] = torch.ops.aten.slice.Tensor(slice_23, 1, 1, 9223372036854775807);  slice_23 = None
        
        # File: /home/neo/anaconda3/lib/python3.9/site-packages/torchvision-0.15.0a0+d95fbaf-py3.9-linux-x86_64.egg/torchvision/models/detection/roi_heads.py:699, code: scores = scores[:, 1:]
        slice_25: f32[1000, 91] = torch.ops.aten.slice.Tensor(getitem_1, 0, 0, 9223372036854775807);  getitem_1 = None
        slice_26: f32[1000, 90] = torch.ops.aten.slice.Tensor(slice_25, 1, 1, 9223372036854775807);  slice_25 = None
        
        # File: /home/neo/anaconda3/lib/python3.9/site-packages/torchvision-0.15.0a0+d95fbaf-py3.9-linux-x86_64.egg/torchvision/models/detection/roi_heads.py:700, code: labels = labels[:, 1:]
        slice_27: i64[1000, 91] = torch.ops.aten.slice.Tensor(expand, 0, 0, 9223372036854775807);  expand = None
        slice_28: i64[1000, 90] = torch.ops.aten.slice.Tensor(slice_27, 1, 1, 9223372036854775807);  slice_27 = None
        
        # File: /home/neo/anaconda3/lib/python3.9/site-packages/torchvision-0.15.0a0+d95fbaf-py3.9-linux-x86_64.egg/torchvision/models/detection/roi_heads.py:703, code: boxes = boxes.reshape(-1, 4)
        clone: f32[1000, 90, 4] = torch.ops.aten.clone.default(slice_24, memory_format = torch.contiguous_format);  slice_24 = None
        _unsafe_view: f32[90000, 4] = torch.ops.aten._unsafe_view.default(clone, [90000, 4]);  clone = None
        
        # File: /home/neo/anaconda3/lib/python3.9/site-packages/torchvision-0.15.0a0+d95fbaf-py3.9-linux-x86_64.egg/torchvision/models/detection/roi_heads.py:704, code: scores = scores.reshape(-1)
        clone_1: f32[1000, 90] = torch.ops.aten.clone.default(slice_26, memory_format = torch.contiguous_format);  slice_26 = None
        _unsafe_view_1: f32[90000] = torch.ops.aten._unsafe_view.default(clone_1, [90000]);  clone_1 = None
        
        # File: /home/neo/anaconda3/lib/python3.9/site-packages/torchvision-0.15.0a0+d95fbaf-py3.9-linux-x86_64.egg/torchvision/models/detection/roi_heads.py:705, code: labels = labels.reshape(-1)
        clone_2: i64[1000, 90] = torch.ops.aten.clone.default(slice_28, memory_format = torch.contiguous_format);  slice_28 = None
        _unsafe_view_2: i64[90000] = torch.ops.aten._unsafe_view.default(clone_2, [90000]);  clone_2 = None
        
        # File: /home/neo/anaconda3/lib/python3.9/site-packages/torchvision-0.15.0a0+d95fbaf-py3.9-linux-x86_64.egg/torchvision/models/detection/roi_heads.py:708, code: inds = torch.where(scores > self.score_thresh)[0]
        gt: b8[90000] = torch.ops.aten.gt.Scalar(_unsafe_view_1, 0.05)
        return (gt, _unsafe_view, _unsafe_view_1, _unsafe_view_2)

FX graph 35

class <lambda>(torch.nn.Module):
    def forward(self, arg0_1: i64[338], arg1_1: f32[90000, 4], arg2_1: f32[90000], arg3_1: i64[90000]):
        # File: /home/neo/anaconda3/lib/python3.9/site-packages/torchvision-0.15.0a0+d95fbaf-py3.9-linux-x86_64.egg/torchvision/models/detection/roi_heads.py:709, code: boxes, scores, labels = boxes[inds], scores[inds], labels[inds]
        index: f32[338, 4] = torch.ops.aten.index.Tensor(arg1_1, [arg0_1]);  arg1_1 = None
        index_1: f32[338] = torch.ops.aten.index.Tensor(arg2_1, [arg0_1]);  arg2_1 = None
        index_2: i64[338] = torch.ops.aten.index.Tensor(arg3_1, [arg0_1]);  arg3_1 = arg0_1 = None
        return (index, index_1, index_2)

gt in FX graph34 is a greater than op. Its output buffer size is 90000 and the expected indexes number with condition True is 345. However, in FX graph35, we find the input size is 338. It seems that the issue is caused by dynamo FX, which generates a wrong input size in condition of the input example size with inequal height and width.

[Valentine233]

Update 11/08:

• I used the debug tool and it showed the issue was more likely in inductor instead of dynamo.
• As FX graph35 got the wrong input size, the issue maybe already exist. So I looked before and found two other size mismatches.

The First Mismatch

FX graph 2 and 3 do the image transformation. The image size is supposed to be (3, 799, 1199) before FX graph2, and turns to (3, 800, 1216) after FX graph3. However, we see (800, 1199) instead of (799, 1199) in FX graph2.

FX graph 2

class <lambda>(torch.nn.Module):
    def forward(self, arg0_1: f32[3, 427, 640]):
        # File: /home/neo/anaconda3/lib/python3.9/site-packages/torchvision-0.15.0a0+d95fbaf-py3.9-linux-x86_64.egg/torchvision/models/detection/transform.py:54, code: image[None],
        unsqueeze: f32[1, 3, 427, 640] = torch.ops.aten.unsqueeze.default(arg0_1, 0);  arg0_1 = None
        
        # File: /home/neo/anaconda3/lib/python3.9/site-packages/torchvision-0.15.0a0+d95fbaf-py3.9-linux-x86_64.egg/torchvision/models/detection/transform.py:53, code: image = torch.nn.functional.interpolate(
        arange: f32[800] = torch.ops.aten.arange.default(800, dtype = torch.float32, device = device(type='cpu'), pin_memory = False)
        alias: f32[800] = torch.ops.aten.alias.default(arange);  arange = None
        arange_1: f32[1199] = torch.ops.aten.arange.default(1199, dtype = torch.float32, device = device(type='cpu'), pin_memory = False)
        alias_1: f32[1199] = torch.ops.aten.alias.default(arange_1);  arange_1 = None
        add: f32[800] = torch.ops.aten.add.Tensor(alias, 0.5);  alias = None
        mul: f32[800] = torch.ops.aten.mul.Tensor(add, 0.53375);  add = None
        sub: f32[800] = torch.ops.aten.sub.Tensor(mul, 0.5);  mul = None
        
        # No stacktrace found for following nodes
        _tensor_constant0 = self._tensor_constant0
        
        # File: /home/neo/anaconda3/lib/python3.9/site-packages/torchvision-0.15.0a0+d95fbaf-py3.9-linux-x86_64.egg/torchvision/models/detection/transform.py:53, code: image = torch.nn.functional.interpolate(
        lift_fresh_copy: f32[] = torch.ops.aten.lift_fresh_copy.default(_tensor_constant0);  _tensor_constant0 = None
        maximum: f32[800] = torch.ops.aten.maximum.default(sub, lift_fresh_copy);  sub = lift_fresh_copy = None
        add_1: f32[1199] = torch.ops.aten.add.Tensor(alias_1, 0.5);  alias_1 = None
        mul_1: f32[1199] = torch.ops.aten.mul.Tensor(add_1, 0.5337781484570475);  add_1 = None
        sub_1: f32[1199] = torch.ops.aten.sub.Tensor(mul_1, 0.5);  mul_1 = None
        
        # No stacktrace found for following nodes
        _tensor_constant1 = self._tensor_constant1
        
        # File: /home/neo/anaconda3/lib/python3.9/site-packages/torchvision-0.15.0a0+d95fbaf-py3.9-linux-x86_64.egg/torchvision/models/detection/transform.py:53, code: image = torch.nn.functional.interpolate(
        lift_fresh_copy_1: f32[] = torch.ops.aten.lift_fresh_copy.default(_tensor_constant1);  _tensor_constant1 = None
        maximum_1: f32[1199] = torch.ops.aten.maximum.default(sub_1, lift_fresh_copy_1);  sub_1 = lift_fresh_copy_1 = None
        floor: f32[800] = torch.ops.aten.floor.default(maximum)
        _to_copy: i64[800] = torch.ops.aten._to_copy.default(floor, dtype = torch.int64);  floor = None
        ceil: f32[800] = torch.ops.aten.ceil.default(maximum)
        
        # No stacktrace found for following nodes
        _tensor_constant2 = self._tensor_constant2
        
        # File: /home/neo/anaconda3/lib/python3.9/site-packages/torchvision-0.15.0a0+d95fbaf-py3.9-linux-x86_64.egg/torchvision/models/detection/transform.py:53, code: image = torch.nn.functional.interpolate(
        lift_fresh_copy_2: f32[] = torch.ops.aten.lift_fresh_copy.default(_tensor_constant2);  _tensor_constant2 = None
        minimum: f32[800] = torch.ops.aten.minimum.default(ceil, lift_fresh_copy_2);  ceil = lift_fresh_copy_2 = None
        _to_copy_1: i64[800] = torch.ops.aten._to_copy.default(minimum, dtype = torch.int64);  minimum = None
        floor_1: f32[1199] = torch.ops.aten.floor.default(maximum_1)
        _to_copy_2: i64[1199] = torch.ops.aten._to_copy.default(floor_1, dtype = torch.int64);  floor_1 = None
        ceil_1: f32[1199] = torch.ops.aten.ceil.default(maximum_1)
        
        # No stacktrace found for following nodes
        _tensor_constant3 = self._tensor_constant3
        
        # File: /home/neo/anaconda3/lib/python3.9/site-packages/torchvision-0.15.0a0+d95fbaf-py3.9-linux-x86_64.egg/torchvision/models/detection/transform.py:53, code: image = torch.nn.functional.interpolate(
        lift_fresh_copy_3: f32[] = torch.ops.aten.lift_fresh_copy.default(_tensor_constant3);  _tensor_constant3 = None
        minimum_1: f32[1199] = torch.ops.aten.minimum.default(ceil_1, lift_fresh_copy_3);  ceil_1 = lift_fresh_copy_3 = None
        _to_copy_3: i64[1199] = torch.ops.aten._to_copy.default(minimum_1, dtype = torch.int64);  minimum_1 = None
        unsqueeze_1: f32[800, 1] = torch.ops.aten.unsqueeze.default(maximum, 1);  maximum = None
        unsqueeze_2: i64[800, 1] = torch.ops.aten.unsqueeze.default(_to_copy, 1);  _to_copy = None
        unsqueeze_3: i64[800, 1] = torch.ops.aten.unsqueeze.default(_to_copy_1, 1);  _to_copy_1 = None
        slice_1: f32[1, 3, 427, 640] = torch.ops.aten.slice.Tensor(unsqueeze, 0, 0, 9223372036854775807)
        slice_2: f32[1, 3, 427, 640] = torch.ops.aten.slice.Tensor(slice_1, 1, 0, 9223372036854775807);  slice_1 = None
        index: f32[1, 3, 800, 1199] = torch.ops.aten.index.Tensor(slice_2, [None, None, unsqueeze_2, _to_copy_2]);  slice_2 = None
        slice_3: f32[1, 3, 427, 640] = torch.ops.aten.slice.Tensor(unsqueeze, 0, 0, 9223372036854775807)
        slice_4: f32[1, 3, 427, 640] = torch.ops.aten.slice.Tensor(slice_3, 1, 0, 9223372036854775807);  slice_3 = None
        index_1: f32[1, 3, 800, 1199] = torch.ops.aten.index.Tensor(slice_4, [None, None, unsqueeze_3, _to_copy_2]);  slice_4 = None
        slice_5: f32[1, 3, 427, 640] = torch.ops.aten.slice.Tensor(unsqueeze, 0, 0, 9223372036854775807)
        slice_6: f32[1, 3, 427, 640] = torch.ops.aten.slice.Tensor(slice_5, 1, 0, 9223372036854775807);  slice_5 = None
        index_2: f32[1, 3, 800, 1199] = torch.ops.aten.index.Tensor(slice_6, [None, None, unsqueeze_2, _to_copy_3]);  slice_6 = None
        slice_7: f32[1, 3, 427, 640] = torch.ops.aten.slice.Tensor(unsqueeze, 0, 0, 9223372036854775807);  unsqueeze = None
        slice_8: f32[1, 3, 427, 640] = torch.ops.aten.slice.Tensor(slice_7, 1, 0, 9223372036854775807);  slice_7 = None
        index_3: f32[1, 3, 800, 1199] = torch.ops.aten.index.Tensor(slice_8, [None, None, unsqueeze_3, _to_copy_3]);  slice_8 = unsqueeze_3 = _to_copy_3 = None
        sub_2: f32[800, 1] = torch.ops.aten.sub.Tensor(unsqueeze_1, unsqueeze_2);  unsqueeze_1 = unsqueeze_2 = None
        
        # No stacktrace found for following nodes
        _tensor_constant4 = self._tensor_constant4
        
        # File: /home/neo/anaconda3/lib/python3.9/site-packages/torchvision-0.15.0a0+d95fbaf-py3.9-linux-x86_64.egg/torchvision/models/detection/transform.py:53, code: image = torch.nn.functional.interpolate(
        lift_fresh_copy_4: f32[] = torch.ops.aten.lift_fresh_copy.default(_tensor_constant4);  _tensor_constant4 = None
        sub_3: f32[800, 1] = torch.ops.aten.sub.Tensor(lift_fresh_copy_4, sub_2);  lift_fresh_copy_4 = None
        sub_4: f32[1199] = torch.ops.aten.sub.Tensor(maximum_1, _to_copy_2);  maximum_1 = _to_copy_2 = None
        
        # No stacktrace found for following nodes
        _tensor_constant5 = self._tensor_constant5
        
        # File: /home/neo/anaconda3/lib/python3.9/site-packages/torchvision-0.15.0a0+d95fbaf-py3.9-linux-x86_64.egg/torchvision/models/detection/transform.py:53, code: image = torch.nn.functional.interpolate(
        lift_fresh_copy_5: f32[] = torch.ops.aten.lift_fresh_copy.default(_tensor_constant5);  _tensor_constant5 = None
        sub_5: f32[1199] = torch.ops.aten.sub.Tensor(lift_fresh_copy_5, sub_4);  lift_fresh_copy_5 = None
        mul_2: f32[1, 3, 800, 1199] = torch.ops.aten.mul.Tensor(index, sub_3);  index = None
        mul_3: f32[1, 3, 800, 1199] = torch.ops.aten.mul.Tensor(index_1, sub_2);  index_1 = None
        add_2: f32[1, 3, 800, 1199] = torch.ops.aten.add.Tensor(mul_2, mul_3);  mul_2 = mul_3 = None
        mul_4: f32[1, 3, 800, 1199] = torch.ops.aten.mul.Tensor(index_2, sub_3);  index_2 = sub_3 = None
        mul_5: f32[1, 3, 800, 1199] = torch.ops.aten.mul.Tensor(index_3, sub_2);  index_3 = sub_2 = None
        add_3: f32[1, 3, 800, 1199] = torch.ops.aten.add.Tensor(mul_4, mul_5);  mul_4 = mul_5 = None
        mul_6: f32[1, 3, 800, 1199] = torch.ops.aten.mul.Tensor(add_2, sub_5);  add_2 = sub_5 = None
        mul_7: f32[1, 3, 800, 1199] = torch.ops.aten.mul.Tensor(add_3, sub_4);  add_3 = sub_4 = None
        add_4: f32[1, 3, 800, 1199] = torch.ops.aten.add.Tensor(mul_6, mul_7);  mul_6 = mul_7 = None
        select: f32[3, 800, 1199] = torch.ops.aten.select.int(add_4, 0, 0);  add_4 = None
        return (select,)

FX graph 3

class <lambda>(torch.nn.Module):
    def forward(self, arg0_1: f32[3, 800, 1199]):
        # File: /home/neo/anaconda3/lib/python3.9/site-packages/torchvision-0.15.0a0+d95fbaf-py3.9-linux-x86_64.egg/torchvision/models/detection/transform.py:240, code: batched_imgs = images[0].new_full(batch_shape, 0)
        full: f32[1, 3, 800, 1216] = torch.ops.aten.full.default([1, 3, 800, 1216], 0, dtype = torch.float32, layout = torch.strided, device = device(type='cpu'), pin_memory = False)
        alias: f32[1, 3, 800, 1216] = torch.ops.aten.alias.default(full);  full = None
        
        # File: /home/neo/anaconda3/lib/python3.9/site-packages/torchvision-0.15.0a0+d95fbaf-py3.9-linux-x86_64.egg/torchvision/models/detection/transform.py:243, code: batched_imgs[i, : img.shape[0], : img.shape[1], : img.shape[2]].copy_(img)
        select: f32[3, 800, 1216] = torch.ops.aten.select.int(alias, 0, 0)
        slice_1: f32[3, 800, 1199] = torch.ops.aten.slice.Tensor(select, 2, 0, 1199);  select = None
        copy_: f32[3, 800, 1199] = torch.ops.aten.copy_.default(slice_1, arg0_1);  slice_1 = arg0_1 = None
        return (alias,)

The Second Mismatch

The following FX graphs choose bounding box indices for the 5 classes according to NMS. Class 1 to class 4 have 1000 current indices and class 5 has 741, 4741 as total. After NMS, the current keeping indices numbers are reduced to 392, 219, 252, 283, 209 for each class, 1355 as total. However, in FX graph21, we get 1370 remaining boxes.

Note: FX graph 15 and 20 have no debug files.

FX graph 13

class <lambda>(torch.nn.Module):
    def forward(self, arg0_1: i64[5], arg1_1: i64[4741]):
        # File: /home/neo/anaconda3/lib/python3.9/site-packages/torchvision-0.15.0a0+d95fbaf-py3.9-linux-x86_64.egg/torchvision/ops/boxes.py:107, code: for class_id in torch.unique(idxs):
        select: i64[] = torch.ops.aten.select.int(arg0_1, 0, 0)
        select_1: i64[] = torch.ops.aten.select.int(arg0_1, 0, 1)
        select_2: i64[] = torch.ops.aten.select.int(arg0_1, 0, 2)
        select_3: i64[] = torch.ops.aten.select.int(arg0_1, 0, 3)
        select_4: i64[] = torch.ops.aten.select.int(arg0_1, 0, 4);  arg0_1 = None
        
        # File: /home/neo/anaconda3/lib/python3.9/site-packages/torchvision-0.15.0a0+d95fbaf-py3.9-linux-x86_64.egg/torchvision/ops/boxes.py:108, code: curr_indices = torch.where(idxs == class_id)[0]
        eq: b8[4741] = torch.ops.aten.eq.Tensor(arg1_1, select);  arg1_1 = None
        return (select_1, select_2, select_3, select_4, eq, select)

FX graph 14

class <lambda>(torch.nn.Module):
    def forward(self, arg0_1: i64[1000], arg1_1: f32[4741, 4], arg2_1: f32[4741]):
        # File: /home/neo/anaconda3/lib/python3.9/site-packages/torchvision-0.15.0a0+d95fbaf-py3.9-linux-x86_64.egg/torchvision/ops/boxes.py:109, code: curr_keep_indices = nms(boxes[curr_indices], scores[curr_indices], iou_threshold)
        index: f32[1000, 4] = torch.ops.aten.index.Tensor(arg1_1, [arg0_1]);  arg1_1 = None
        index_1: f32[1000] = torch.ops.aten.index.Tensor(arg2_1, [arg0_1]);  arg2_1 = arg0_1 = None
        return (index, index_1)

FX graph 16

class <lambda>(torch.nn.Module):
    def forward(self, arg0_1: i64[1000], arg1_1: f32[4741, 4], arg2_1: f32[4741]):
        # File: /home/neo/anaconda3/lib/python3.9/site-packages/torchvision-0.15.0a0+d95fbaf-py3.9-linux-x86_64.egg/torchvision/ops/boxes.py:109, code: curr_keep_indices = nms(boxes[curr_indices], scores[curr_indices], iou_threshold)
        index: f32[1000, 4] = torch.ops.aten.index.Tensor(arg1_1, [arg0_1]);  arg1_1 = None
        index_1: f32[1000] = torch.ops.aten.index.Tensor(arg2_1, [arg0_1]);  arg2_1 = arg0_1 = None
        return (index, index_1)

FX graph 17

class <lambda>(torch.nn.Module):
    def forward(self, arg0_1: i64[1000], arg1_1: f32[4741, 4], arg2_1: f32[4741]):
        # File: /home/neo/anaconda3/lib/python3.9/site-packages/torchvision-0.15.0a0+d95fbaf-py3.9-linux-x86_64.egg/torchvision/ops/boxes.py:109, code: curr_keep_indices = nms(boxes[curr_indices], scores[curr_indices], iou_threshold)
        index: f32[1000, 4] = torch.ops.aten.index.Tensor(arg1_1, [arg0_1]);  arg1_1 = None
        index_1: f32[1000] = torch.ops.aten.index.Tensor(arg2_1, [arg0_1]);  arg2_1 = arg0_1 = None
        return (index, index_1)

FX graph 18

class <lambda>(torch.nn.Module):
    def forward(self, arg0_1: i64[1000], arg1_1: f32[4741, 4], arg2_1: f32[4741]):
        # File: /home/neo/anaconda3/lib/python3.9/site-packages/torchvision-0.15.0a0+d95fbaf-py3.9-linux-x86_64.egg/torchvision/ops/boxes.py:109, code: curr_keep_indices = nms(boxes[curr_indices], scores[curr_indices], iou_threshold)
        index: f32[1000, 4] = torch.ops.aten.index.Tensor(arg1_1, [arg0_1]);  arg1_1 = None
        index_1: f32[1000] = torch.ops.aten.index.Tensor(arg2_1, [arg0_1]);  arg2_1 = arg0_1 = None
        return (index, index_1)

FX graph 19

class <lambda>(torch.nn.Module):
    def forward(self, arg0_1: i64[741], arg1_1: f32[4741, 4], arg2_1: f32[4741]):
        # File: /home/neo/anaconda3/lib/python3.9/site-packages/torchvision-0.15.0a0+d95fbaf-py3.9-linux-x86_64.egg/torchvision/ops/boxes.py:109, code: curr_keep_indices = nms(boxes[curr_indices], scores[curr_indices], iou_threshold)
        index: f32[741, 4] = torch.ops.aten.index.Tensor(arg1_1, [arg0_1]);  arg1_1 = None
        index_1: f32[741] = torch.ops.aten.index.Tensor(arg2_1, [arg0_1]);  arg2_1 = arg0_1 = None
        return (index, index_1)

FX graph 21

class <lambda>(torch.nn.Module):
    def forward(self, arg0_1: i64[1370], arg1_1: f32[4741]):
        # File: /home/neo/anaconda3/lib/python3.9/site-packages/torchvision-0.15.0a0+d95fbaf-py3.9-linux-x86_64.egg/torchvision/ops/boxes.py:112, code: return keep_indices[scores[keep_indices].sort(descending=True)[1]]
        index: f32[1370] = torch.ops.aten.index.Tensor(arg1_1, [arg0_1]);  arg1_1 = None
        sort = torch.ops.aten.sort.default(index, -1, True);  index = None
        getitem: f32[1370] = sort[0]
        getitem_1: i64[1370] = sort[1];  sort = None
        index_1: i64[1370] = torch.ops.aten.index.Tensor(arg0_1, [getitem_1]);  arg0_1 = getitem_1 = None
        return (index_1,)

[Valentine233]

Update 11/08:
Code: /home/neo/anaconda3/lib/python3.9/site-packages/torchvision-0.15.0a0+d95fbaf-py3.9-linux-x86_64.egg/torchvision/models/detection/transform.py

In line 50, the value of scale_factor is 1.873536229133606, with float type. In line 53, origin image size is [3, 427, 640]. The output size with float type is [799.99996984, 1199.06318665] = [3, 427, 640] * 1.873536229133606. The output image size is [799, 1199] with eager mode and [800, 1199] with inductor. However, when I modify line 50 with scale_factor = 1.873536229133606, the output size with inductor is the same as eager mode and the model could pass.

Corresponding graph: FX graph2

[Valentine233]

Update 11/09:
The size mismatch between eager mode and inductor is caused by data type leading a precision problem.

Let's see the code of _resize_image_and_masks above.

In line 45, since min_size and max_size are float32, we got scale as tensor(1.8735, dtype=torch.float32), keeping only 7 decimal digits. However, as scale_factor is float, aka. float64 in Python, the numbers after 7th digit are undefined. Hence, eager mode got scale_factor of 1.873536229133606 and inductor got scale_factor of 1.8735363483428955. For the output shape, eager mode got 799 = int(floor(1.873536229133606*427)) and inductor got 800 = int(floor(1.8735363483428955*427)).

To fix the problem, lines 43 and 44 should be modified as followed.

min_size = torch.min(im_shape).to(dtype=**torch.float64**)
max_size = torch.max(im_shape).to(dtype=**torch.float64**)

[Valentine233]

Update 11/10:
According to the previous analysis, eager mode got scale of 1.873536229133606 and inductor got scale of 1.8735363483428955. It is found that the difference is caused by the data type when calculating scale.

Calculation formula: scale = torch.min(self_min_size / min_size, self_max_size / max_size)

Data types in eager mode: (float64 self_min_size, float32 min_size, float64 self_max_size, float32 max_size) -> float32 scale
Data types with inductor: (float32 self_min_size, float32 min_size, float32 self_max_size, float32 max_size) -> float32 scale

The root cause of the issue is that the inductor takes self_min_size and self_max_size as float32 when deducing the data types of constants.

The related code is as follows.
pytorch/torch/_inductor/lowering.py:264

The data type of the constant (self_min_size and self_max_size) would be the same as the tensor (min_size and max_size) with which it multiplies.

[Valentine233]

Conclusion 11/11:

This is a precision issue between eager mode and inductor, caused by their different inferences for constant's data type. Eager mode treats all float constants in python as double, while inductor makes the constant's data type the same as other input tensors of the operator.

In this case, the core code is as follows.
mul: f32[] = torch.ops.aten.mul.Tensor(reciprocal, 800.0);
Input1 reciprocal is a float32 tensor. Input2 800.0 is a float constant in python. Output mul is a float32 tensor.
Eager mode turns 800.0 to a float64 tensor, while inductor turns 800.0 to a float32 tensor, leading the final result size mismatch problem.

We will discuss what to do and how to fix this later.

[ngimel]

But 800.0 is exactly representable as float32, so why is result different? is it different division precision in inductor and eager, regardless of input data?

[jgong5]

But 800.0 is exactly representable as float32, so why is result different? is it different division precision in inductor and eager, regardless of input data?

Yep, we are still investigating... The dumped FX graph seems correct (keep the same semantics as eager). Maybe the generated C++ code is wrong somewhere.

[Valentine233]

Problem

In eager mode, all float constants are converted to double tensors. In this case, 800.0's data type is converted to torch.float64.
Code: pytorch/aten/src/Aten/ScalarOps.h:39

For inductor, the constant's data type is decided by other input tensors. In this case, 800.0's data type is converted to torch.float32.
Code: pytorch/torch/_inductor/lowering.py:279

Fix

To fix the problem, we could modify the code above in lowering like this:

Besides, the operator output should keep its type, so the cpp codegen also need a modification.
Code: debug.1/output_code.py

In line 62, with tmp2 float and tmp3 double, tmp4 would be double instead of float. Hence, we need to know the type of tmp4 and cast it to float after the multiplication operator. It has to be figured out how to get the output types.

Next

1. For the codegen modification, we need a further discussion.
2. Due to the conversion from float32 to float64, fixing the issue may lead to a performance regression.
3. The inconsistence of inputs' data types in binary operators may cause problems for vectorizations.

[jgong5]

@ngimel I guess promote_constants is used by both CPU and CUDA. Do you know if vision_maskrcnn is working well on CUDA?

[ngimel]

I don't know if vision_maskrcnn works on cuda, but I still don't understand what difference the dtype of 800 makes, whether it's float or double it's still exactly representable, and it shouldn't affect output dtype. Can you provide minimum repro of inductor producing different results from eager, with exact input values?