[inductor] TypeError: function takes exactly 16 arguments (13 given)

[typedfemale]

I have a pretty odd one. Minimal repo below:

import torch
from torch import tensor, device
import torch.fx as fx
from torchdynamo.testing import rand_strided
from math import inf
from torch.fx.experimental.proxy_tensor import make_fx

# torch version: 1.14.0.dev20221009
# torch cuda version: 11.7
# torch git version: 0dbefb2414417e80371ef3d8224404d4a522f86e


# CUDA Info:
# nvcc: NVIDIA (R) Cuda compiler driver
# Copyright (c) 2005-2022 NVIDIA Corporation
# Built on Wed_Jun__8_16:49:14_PDT_2022
# Cuda compilation tools, release 11.7, V11.7.99
# Build cuda_11.7.r11.7/compiler.31442593_0

# GPU Hardware Info:
# NVIDIA A100-SXM4-40GB : 1


from torch.nn import *
class Repro(torch.nn.Module):
    def __init__(self):
        super().__init__()



    def forward(self, arg0_1, new_zeros_1):
        slice_scatter = torch.ops.aten.slice_scatter.default(new_zeros_1, arg0_1, 2, 0, 2048);  new_zeros_1 = arg0_1 = None
        return (slice_scatter,)

args = [((16, 128, 2048), (262144, 2048, 1), torch.float32, 'cuda'), ((16, 128, 2112), (270336, 2112, 1), torch.float32, 'cuda')]
args = [rand_strided(sh, st, dt, dev) for (sh, st, dt, dev) in args]
mod = make_fx(Repro())(*args)

from torchinductor.compile_fx import compile_fx_inner
from torchdynamo.debug_utils import same_two_models

compiled = compile_fx_inner(mod, args)
compiled(*args)

Needed for fast state space models 😄

[ngimel]

Can't repro on master

[typedfemale]

I'm really stuck on this one. My exact steps are as follows (and I've reproduced it twice, once using conda, once with pip):

1. I create a new VM on GCP with "Debian 10 based Deep Learning VM with M97 - Base CUDA 11.3, Deep Learning VM Image with CUDA 11.3 preinstalled"
2. It asks if you want to install drivers, agree.
3. Install torch. (pip3 install --pre torch --extra-index-url https://download.pytorch.org/whl/nightly/cu117 )
4. Install triton (pip install -U --pre triton)
5. Install torchdynamo (pip install -U "git+https://github.com/pytorch/torchdynamo")
6. Run repo.py

I'm at a complete loss what I could be missing at this point, I've tried many many times.

[typedfemale]

See this colab notebook as well: https://colab.research.google.com/drive/1k-5TExj3m2KMNN0evUTOicfYnjRd6cxu#scrollTo=AKf0TyWhMN3d 😭

[desertfire]

This might relate to a triton version issue. You can give

torchdynamo/Makefile

Line 16 in 3f92a09

TRITON_VERSION ?= d3c925db8a81ca74f14680876b9311e7d079c5a1

this pinned triton version a try.

[typedfemale]

Yes, potentially it is a triton issue. Sadly, using that triton commit failed for a different reason:

[<ipython-input-13-dd3d98fb795d>](https://localhost:8080/#) in <module>
     40 from torchdynamo.debug_utils import same_two_models
     41 
---> 42 compiled = compile_fx_inner(mod, args)
     43 compiled(*args)

12 frames
[/usr/local/lib/python3.7/dist-packages/torchinductor/codegen/triton.py](https://localhost:8080/#) in codegen_kernel(self, name)
    921 
    922     def codegen_kernel(self, name=None):
--> 923         from triton import next_power_of_2
    924 
    925         code = IndentedBuffer()

ImportError: cannot import name 'next_power_of_2' from 'triton' (unknown location)```

[jansel]

I saw this too with the latest Triton. I think something upstream changed in the new runtime.

[jansel]

@typedfemale this one might fix it

pip uninstall triton
pip install -U "git+https://github.com/openai/triton@d3c925db8a81ca74f14680876b9311e7d079c5a1#subdirectory=python"

[typedfemale]

@jansel I tried it here, but I get a different error (I believe that commit is the same one @desertfire mentioned)...

[jansel]

@jansel I tried it here, but I get a different error (I believe that commit is the same one @desertfire mentioned)...

Ah too bad. I'm guessing from your PR you go it working though.

[typedfemale]

Just waiting on #1585 now since it seems like I broke caching (I just tested that branch and I think they have the same issue I do) 😅

[typedfemale]

#1585 is merged