[Feature Request] Calculating FLOPs for computational graph operations

[MG2033]

Feature Request: Please consider adding a floating point operations calculator for computational graph operations.

We'd like to use it for the deep learning models.

cc @ezyang @gchanan @zou3519 @bdhirsh @heitorschueroff @VitalyFedyunin @ngimel

[soumith]

@cpuhrsch this is something you were interested to have as well right? also @shubho

[cpuhrsch]

@soumith, this will be important, but it's low priority for now as we're still experimenting outside of pytorch. We're setting something up similar to this, but not as part of any framework for now. Maybe we can port that code to pytorch later on, but I can't promise anything.

[shubho]

Would be nice to track which networks do well on which processors - and also roll up datacenter level metrics. @cpuhrsch are you building a tool outside of PyTorch that will take in a model specification and emit these numbers for every op?

[cpuhrsch]

@shubho for now we're doing this one op at a time. For example, we're building it out far enough to optimize a function such as torch.sum, but within C++ only. Presumably, as we work our way through this, a tool that will create these (and many more) metrics per op for a model could emerge. For now that'd be over-engineering our tooling.

[ahirner]

I've come across this script to count ops and params for basic layers. It's reasonably advanced (most 2D convs and pooling). Maybe it compliments your porting efforts @cpuhrsch.

[warmspringwinds]

If someone still needs this, we wrote a small script to do that:

https://github.com/warmspringwinds/pytorch-segmentation-detection/blob/master/pytorch_segmentation_detection/utils/flops_benchmark.py

Example of usage:

https://github.com/warmspringwinds/pytorch-segmentation-detection/blob/d5df5e066fe9c6078d38b26527d93436bf869b1c/pytorch_segmentation_detection/recipes/pascal_voc/segmentation/flops_counter.ipynb

[sovrasov]

Also I've improved the above mentioned scripts to support grouped convolutions and several other layers:
https://github.com/sovrasov/flops-counter.pytorch

[laoreja]

We can compute FLOPs of conv layers by hand or by existing repos easily.
But is there a way to compute FLOPs for non-traditional operations like topk, sparse sum and indexing, etc.?
Thanks!

[cpuhrsch]

@zheng-xq - Is this something that could be of interest to you?

[ttumiel]

Came across this library which also counts number of parameters and FLOPs used by a model given an input.

[matyasfodor]

I came across this repo: https://github.com/ceykmc/pytorch_model_summary AFAIK it's not distributed on pypi, but it gives you a pretty detailed summary.

[matyasfodor]

I just found this: Autograd profiler I wonder if this can be used for calculating the FLOPs and memory usage. I tried the chrome trace export, but I'm not yet sure how to get something useful out of it.

[bhack]

Just to collect another solution https://github.com/Tramac/torchscope.

But it could be better if there will be an off the shelf solution directly in pytorch.

[ezyang]

@pietern says, this has essentially already been implemented on the Caffe2 side; there's a lot of overlap here.

[cpuhrsch]

cc @ilia-cher

[ruiyuanlu]

so ... any plan to provide such calculator in the next version pytorch? It would be nice to have an official tool to calculate FLOPS. 3rd party implementations might not support the extension of pytorch ops properly.

[ruiyuanlu]

well you know, something like

tf.profiler.profile(g, run_meta=run_meta, cmd='op', options=opts)

would be very helpful.

[harryherold]

@ruiyuanlu Is it also working with Keras?

[matkalinowski]

Are there any plans on implementing that idea?

[ezyang]

Bumping priority so we don't forget about this.

[mruberry]

cc @ngimel, who's been doing work in this area

[ilia-cher]

cc. @xuzhao9 is working on adding flops into profiler output

[ngimel]

#46506 is the profiler PR.

[ppwwyyxx]

In fvcore we created the first flop counter which has two following features together:

• counts flops at operator level (thus more accurate than existing module-level counters)
• provide module-level aggregations (thus more friendly to users)

We gave an introduction of it at this documentation: https://github.com/facebookresearch/fvcore/blob/master/docs/flop_count.md

[Jackiexiao]

a simple compare with different flops count lib, in short, I recommand fvcore

https://gist.github.com/Jackiexiao/2b053cd52d977d86e07d664688c0a7ee

[nkalpakis21]

i see this has been open for awhile. is there trouble getting a dev to fix this issue? what is blocking this for so long

[zou3519]

Can we close this @Chillee @albanD? afaict this has been implemented (https://gist.github.com/Chillee/07b36672a0ca2d1280e42b8d10f23174)

[Chillee]

Yes, we have implemented a flop counter in core.

from torch.utils.flop_counter import FlopCounterMode

from torchvision.models import resnet18
model = resnet18().cuda().half()
inp = torch.randn(128, 3, 224, 224, device='cuda', dtype=torch.half)

with FlopCounterMode():
    model(inp).sum().backward()