[Bug] Project-level rules don't show up in IDE

[Lakitna]

What happened?

When using Robocop in an IDE, the new unused-keyword rule does not show up.

I dug a bit deeper, this is caused by how both robocorp/robotframework-lsp and robotcodedev/robotcode call Robocop. The relevant lines are:

• robocorp/robotframework-lsp
• robotcodedev/robotcode

Both language servers call run_check. Because of this, the newer run_project_checks is not executed by the language servers.

If you run Robcop from the command line, the run_checks function ensures that both run_check and run_poject_checks are called.

What command/code did you try to run?

I used VSCode with robocorp/robotframework-lsp, then ran Robocop through the language server.

What is the full error message?

N/A. The rule is not executed

What did you expect to happen instead?

I expect the output in my IDE to be the same as the output in the CLI. Otherwise, the usefulness of project-level rules is greatly diminished.

I recognize that this was probably a deliberate choice made for performance reasons. I'm making this issue in part to discuss where this should be fixed. I have not found this issue in either of the language server repositories.

Operating System

Windows

Robocop version

5.4.0

[bhirsz]

It wasn't entirely because of the performance reason - usually project checks are disabled by default so it would not affect users if I were to implement it in old method. The main reason behind current implementation is that I wanted to ensure that we run project checks last - after scanning all other files. While scanning other files we gather statistics/data (it's done whenever we call something with run_check) and then run_project_checks scans accumulated data and optionally finds issues.

One solution would be simply adding run_project_checks to both plugins source code. But it will be far from ideal because "project check" will transform into "scan this file with normal checks and then scan with project checks" - since those plugins scan each file separetely. Current only existing project check (unused keyword) will still work because it's in alpha state and only finds private / suite keywords but it may not work in the future after update.

I need to think about it, possibly other option of calling robocop on the project level in those plugins would be best. It could be also more optimal since plugins have access to project 'root' while robocop usually tries to guess it.

[Lakitna]

With the whole 6.0.0/robotidy migration thing. Maybe this should be solved in the new architecture?

[bhirsz]

Yeah, I want to try it - it's already possible to do it in current Robocop but I would prefer to implement it in 6.0 (let's call it robocop2 for now) as I will be doing a lot of other breaking changes, updates to how plugins use our tool etc so it can be done together

[bhirsz]

I'm going through issues preparing for 6.0 release. We're yet to integrate with RobotCode (I will most likely do not do it for other plugin which is no longer maintained -> RobotCode recently got Pycharm support as well).

The main issue remains: IDE plugins don't run robocop project-wide, then take one single file and gather diagnostics only for it. I can simply run them after 'normal' checker finish, but it will most likely fail if given project checker requires gathering more data.

I may change my approach to whole project-checker instead. Maybe it should run and gather files on its own, so running it on file x.robot will try to find all files in project (determined by .git repository) anyway. Then even if someone runs robocop with single file, it will use whole project for project checker.

Other approach is to accept it as known limitation, and if someone only wants issues from single file then project-level rules will not appear.

[Lakitna]

I would disable a rule that is not always used because it can be confusing. If there are 0 issues in my IDE, I expect 0 issues in the CLI (and vice versa).

It may be good to not try to support project-level checkers at all. A clear scope can be a great feature.

Project-level checkers may work better in a separate project. I've been working on something myself that's tightly scoped to finding unused things in projects: https://github.com/Lakitna/robotframework-find-unused (It's functional and mostly bug-free, but still needs work before I can really advise everyone to use it)

Or, if you don't want to spin project-level checkers into their own thing, maybe it works better as it's own command (argument)?

robocop check --project
robocop checkproject

[bhirsz]

Thats a very good take. Initially I have created it separately as Sherlock project. Which was a lot more advanced that what robocop is currently doing for project checker but still missing various important parts. Then maintaining it as seperate project was too much time consuming so I have decided to move some of it to Robocop.. So we are at full circle :).

But I agree that Robocop should produce the same results and in this case we cant run something else for cli. I like idea of having separate command. For example robocop analyze (similar as 'ruff analyze'). Or even reuse project name and call it robocop sherlock (though it starts to sound cringe..).

At first I will try to move it to separete command. Current architecture make it easy. Not sure if I can have everything ready for 6.0 but at least I will not run it with other checks as first step.