[RFC] Type checking to python code

Subprojects LLDB
1

Background

In lldb, we have a minimum python version of 3.8, if python support is enabled.

Since python 3.5 it has been possible to add type annotations to code and the new ‘typing’ module was introduced.

I have been working on improving the unit tests for lldb-dap and while working on that, I found its helpful to have additional type information. This can help in the IDE by helping me ensure the value or structure or parameters of a function are correct. Here is an example PR of the kinds of type annotations I have been adding.

However by default the type annotations are not going to do much (if anything).

In python, you need to run a static type checker to validate the types. There is some support for runtime type assertions using typing.assert_type, (EDIT: assert_type has no runtime side effects, its only a hint to the type checker) otherwise you need to invoke a tool like mypy or pyright. Other type checkers exist, but I believe those are the two most popular choices.

Proposal

At the moment, any python code annotated with types is not validated as part of the CI. This means that the types could regress over time.

We could address this in a few ways:

  • Add a pre-commit hook to run the type checker, like the linter checks
  • Add a pre-merge hook to run the type checker, like the linter checks
  • Run the type checker before we run the python based tests

Additionally, if we wanted to run a type checker, we would also need to pick which type checker we’d like to use. Like I mentioned previously, mypy and pyright are two popular choices but others exist.

I think mypy is a good option because its supported by the python core team and is a fairly common choice.

I also think it would be worth adding the type_extensions library to the lldb/test/requirements.txt since it includes back ports of new types for older versions of python.

Running mypy

At the moment, we don’t have a config setup specifically for running a type checker, but you can run the files directly like:

$ time mypy --python-version 3.8 \
  lldb/packages/Python/lldbsuite/**/*.py \
  lldb/test/API/tools/lldb-dap/**/*.py
... a couple of warnings / errors from existing code
mypy --python-version 3.8 lldb/packages/Python/lldbsuite/**/*.py   0.66s user 0.18s system 66% cpu 1.270 total

On my machine, an M1 Max MBP, it takes ~1.5s to run the type checker on the lldb unit tests.

Issue tracker link: [lldb] Add a CI step to run a python type checker · Issue #141877 · llvm/llvm-project · GitHub

3 Likes
2

I like the idea of supporting typing where it benefits us. The PR you linked is a good example of where this adds value and help catch mistakes. I can also imagine a bunch of places where it doesn’t make sense and dynamic typing is totally fine (and static types would just add an additional burden). If we’re all on the same page about that, let’s add a snippet to the testing page.

As for the proposal to enforce this, my vote would be to do the same as the formatters and make this a pre-commit PR check. We can add a convenience CMake target (something like check-lldb-types) that runs the types checker on our test suite at desk. I imagine in CI we don’t want to go through the CMake configuration stage.

I don’t have an opinion on mypy vs pyright. I use the latter as my Python LSP server, but that’s pretty much orthogonal.

3

Over on the Clang side, we’re slowly adding type annotations to libclang’s Python bindings. This compelled us to look closely at this layer of our API and find and fix several bugs.

@JannickKremer Any thoughts here?

4

100% in favor of type-checking in the CI!

That was also among my plans for the libclang Python bindings in the future. A few concerns and open questions that I’d like to raise:

  1. the main issue I see is how to deal with incomplete type hints and what the stated goal should be. As Endill mentioned, we are currently in the process of adding annotations for the libclang Python bindings, but haven’t added CI checks here yet. That’s because it is difficult to define a meaningful check for incomplete type hints. Is any PR that doesn’t increase the current number of typing errors a pass? Should the project have 0 type errors? Should it pass a strict type check?
  2. The answers to the above question also depend heavily on the needs of the subproject, as JDevlieghere said. The benefits of type annotations are both internal (they serve as additional documentation and help find bugs) and external (better IDE support for downstream users). The goal with libclang’s Python bindings, for example, is to eventually pass a strict type check, because we want users of the bindings to have better support in their IDE by annotating all the interfaces. In comparison, your project of annotating tests sounds like it only needs the internal benefits, and thus wouldn’t gain much from a strict pass.
  3. As a result, rather than an LLVM-wide type-checking job, it might be preferable to have a job that can be added on a per-project basis, or with different parameters per folder or similar, so every project can decide if/when they are ready and to what extent they want to integrate type-checking in their CI.

Aside from that, I’d vote for mypy but that’s mostly because that’s what I’ve used and contributed to before.

5

I think it would be good to define a config that we can use opt in as we work on areas of the code base. In mypy you need to use the --check-untyped-defs to check all declarations, otherwise it will only check functions that have type annotations.

Here is a prototype init file for lldb with a few error codes disabled:

$ cat lldb/.mypy.ini
[mypy]
python_version = 3.8
files = lldb/packages/Python/lldbsuite/**/*.py,lldb/test/API/tools/lldb-dap/**/*.py
warn_no_return = True
warn_redundant_casts = True
warn_return_any = True
warn_unreachable = True
warn_unused_ignores = True

[mypy-lldb]
disable_error_code = import-not-found,no-redef

[mypy-dap_server]
allow_redefinition_new = True
local_partial_types = True

To run this, use:

$ MYPYPATH=`<build>/bin/lldb -P` mypy --config-file=lldb/.mypy.ini

Right now, we have the following errors:

lldb/packages/Python/lldbsuite/support/gmodules.py:6: error: Need type annotation for "GMODULES_SUPPORT_MAP" (hint: "GMODULES_SUPPORT_MAP: Dict[<type>, <type>] = ...")  [var-annotated]
lldb/packages/Python/lldbsuite/test/lldbtest_config.py:13: error: Need type annotation for "channels" (hint: "channels: List[<type>] = ...")  [var-annotated]
lldb/packages/Python/lldbsuite/test/configuration.py:29: error: Need type annotation for "skip_categories" (hint: "skip_categories: List[<type>] = ...")  [var-annotated]
lldb/packages/Python/lldbsuite/test/configuration.py:31: error: Need type annotation for "xfail_categories" (hint: "xfail_categories: List[<type>] = ...")  [var-annotated]
lldb/packages/Python/lldbsuite/test/configuration.py:33: error: Need type annotation for "failures_per_category" (hint: "failures_per_category: Dict[<type>, <type>] = ...")  [var-annotated]
lldb/packages/Python/lldbsuite/test/configuration.py:56: error: Need type annotation for "settings" (hint: "settings: List[<type>] = ...")  [var-annotated]
lldb/packages/Python/lldbsuite/test/configuration.py:70: error: Need type annotation for "filters" (hint: "filters: List[<type>] = ...")  [var-annotated]
lldb/packages/Python/lldbsuite/test/configuration.py:124: error: Need type annotation for "all_tests" (hint: "all_tests: Set[<type>] = ...")  [var-annotated]
lldb/packages/Python/lldbsuite/test/configuration.py:138: error: Need type annotation for "enabled_plugins" (hint: "enabled_plugins: List[<type>] = ...")  [var-annotated]
lldb/packages/Python/lldbsuite/test/decorators.py:1107: error: "list" is not subscriptable, use "typing.List" instead  [misc]
lldb/packages/Python/lldbsuite/test/decorators.py:1114: error: "None" has no attribute "lower"  [attr-defined]
lldb/packages/Python/lldbsuite/test/tools/lldb-server/lldbgdbserverutils.py:986: error: Module has no attribute "WinDLL"  [attr-defined]
lldb/packages/Python/lldbsuite/test/tools/lldb-server/lldbgdbserverutils.py:1105: error: Name "Pipe" already defined on line 1046  [no-redef]
lldb/test/API/tools/lldb-dap/variables/TestDAP_variables.py:200: error: Unsupported target for indexed assignment ("object")  [index]
lldb/test/API/tools/lldb-dap/variables/TestDAP_variables.py:216: error: Need type annotation for "varref_dict" (hint: "varref_dict: Dict[<type>, <type>] = ...")  [var-annotated]
lldb/test/API/tools/lldb-dap/variables/TestDAP_variables.py:352: error: Value of type "object" is not indexable  [index]
lldb/test/API/tools/lldb-dap/variables/TestDAP_variables.py:353: error: Value of type "object" is not indexable  [index]
lldb/test/API/tools/lldb-dap/variables/TestDAP_variables.py:383: error: Value of type "object" is not indexable  [index]
lldb/test/API/tools/lldb-dap/variables/TestDAP_variables.py:384: error: Value of type "object" is not indexable  [index]
lldb/test/API/tools/lldb-dap/variables/TestDAP_variables.py:385: error: Value of type "object" is not indexable  [index]
lldb/test/API/tools/lldb-dap/variables/TestDAP_variables.py:391: error: Value of type "object" is not indexable  [index]
lldb/test/API/tools/lldb-dap/variables/TestDAP_variables.py:565: error: Need type annotation for "expr_varref_dict" (hint: "expr_varref_dict: Dict[<type>, <type>] = ...")  [var-annotated]
lldb/test/API/tools/lldb-dap/variables/TestDAP_variables.py:566: error: "Collection[str]" has no attribute "items"  [attr-defined]
lldb/test/API/tools/lldb-dap/variables/TestDAP_variables.py:672: error: Incompatible types in assignment (expression has type "Tuple[Dict[str, Dict[str, List[str]]]]", target has type "Dict[str, Dict[str, str]]")  [assignment]
Found 24 errors in 6 files (checked 98 source files)

Which I think is a reasonable set of errors we could address by either disabling specific warnings in specific files or fixing the files.

Yea, the lldb use case is mostly for test development. At the moment, it appears that swig does not support type annotations on the generated output. There is an open issue for adding support to swig though.

6

Sounds good to me!
Note that when I say “strict type check”, I refer to mypy’s --strict flag, which implies all optional checks.
Definitely agree that a flexible, detailed per-project config would be desirable (though not necessarily in the very first step)

7

I got a basic prototype of this integration working by adding this to the lldb dotest.py script.

With this patch, if you use -DLLDB_PYTHON_TESTS_CHECK_TYPES:BOOL=true when you run cmake then the dotest.py script will run mypy on the test prior to running the test.

If you run an individual test with dotest.py with this configuration you’ll see the mypy report in stdout like:

$ python3 llvm-project/lldb/test/API/dotest.py \
  -u CXXFLAGS \
  -u CFLAGS \
  --out-of-tree-debugserver \
  --env LLVM_LIBS_DIR=lldb-build/./lib \
  --env LLVM_INCLUDE_DIR=lldb-build/include \
  --env LLVM_TOOLS_DIR=lldb-build/./bin \
  --libcxx-include-dir lldb-build/include/c++/v1 \
  --libcxx-library-dir lldb-build/lib \
  --arch arm64 \
  --build-dir lldb-build/lldb-test-build.noindex \
  --lldb-module-cache-dir lldb-build/lldb-test-build.noindex/module-cache-lldb/lldb-api \
  --clang-module-cache-dir lldb-build/lldb-test-build.noindex/module-cache-clang/lldb-api \
  --executable lldb-build/./bin/lldb \
  --compiler lldb-build/./bin/clang \
  --dsymutil lldb-build/./bin/dsymutil \
  --make /usr/bin/make \
  --llvm-tools-dir lldb-build/./bin \
  --lldb-obj-root lldb-build/tools/lldb \
  --lldb-libs-dir lldb-build/./lib \
  --framework lldb-build/bin/LLDB.framework \
  --cmake-build-type RelWithDebInfo \
  --check-types \
  llvm-project/lldb/test/API/tools/lldb-dap/io -p TestDAP_io.py
lldb version 21.0.0git ([email protected]:ashgti/llvm-project.git revision c93fc07792ca4d2e2a082fe3e69aaa27ddd0c83e)
  clang revision c93fc07792ca4d2e2a082fe3e69aaa27ddd0c83e
  llvm revision c93fc07792ca4d2e2a082fe3e69aaa27ddd0c83e
Skipping the following test categories: ['libstdcxx', 'dwo', 'llgs', 'fork']
Success: no issues found in 1 source file

Ran 5 tests in 1.961s

OK

Adjusting the file to include an error you will see:

lldb/test/API/tools/lldb-dap/io/TestDAP_io.py:59: error: Incompatible types in assignment (expression has type "int", variable has type "str")  [assignment]
Found 1 error in 1 file (checked 1 source file)

Ran 5 tests in 1.824s

OK

I’m not failing the test if the file does not pass the type check at the moment. We could consider adding that in the future though.

I’m open to suggestions if anyone has any other ideas on how we can run this integration.

8

I think we should at least have a mode where lldb-dotest does not fail if the type checking fails. When you are initially writing a test, lldb-dotest is the most convenient way to iteratively develop it. This is particularly useful with the -d flag when you’re using the test you’ve written to get the feature to work. But in that process you are often adding little utility functions that may not even survive in the final test, more of the “How the heck did we get here” detectors…

Having to be pedantic about type checking during the development of the test would be annoying.

9

I can update the --check-types flag to take a parameter to control the strictness. Maybe something like --check-types=error (causes an error if type checking fails) or --check-types=warn (only reports any errors without failing the test).

EDIT:

Updated my branch with this approach: [lldb] Prototyping type checking python files when running tests. · ashgti/llvm-project@85739f3 · GitHub

When building with cmake, you can use -DLLDB_PYTHON_TESTS_CHECK_TYPES:STRING=strict|warn. It defaults to disabled.

When running with dotest.py you can use the flag --check-types strict|warn or not include the flag at all and it will skip running mypy.

10

In my first iteration at implementing this, I added the flag to lldb’s dotest.py script, but maybe I should be looking into llvm-lit instead.

I’ll take a little time to see if I can find a reasonable way to integrate into lit instead and update my branch.

11

Reading into lit a bit more, I don’t think this makes sense to have in lit. lit isn’t aware of the implementation details of the tests. So, for lldb’s tests right now at least, I think it makes the most sense for this to remain as part of dotest.py for now.