So far, I have worked with traditional web frameworks such as Selenium, Cucumber, and Cypress and there are abundant resources that helped me get started.
In our case, we are building an IntelliJ plugin, and testing it end-to-end was challenging. I couldn’t find an easy-to-get-started guide that explained automated UI tests for IDE plugins. Hope that this post serves the purpose.
Unlogged has two components: the Unlogged SDK (this adds probes to the user code in compile time and logs the code execution) and an IntelliJ plugin. TLDR: The SDK writes the logs, and the IntelliJ plugin scans those logs.
Here is how the plugin works from a developer's point of view:
Although, I created a complete automation mimicking all the steps above, I want to focus only on automating the UI Interaction part of the IntelliJ UI interface.
I came across the Remote Robot tool from IntelliJ. This tool seems well-suited for IntelliJ plugins. It allows you to simulate user actions, such as key presses and mouse movements.
It goes a step further, enabling us to interact with IntelliJ's APIs, offering a way to mimic a real world use case. This coupled with reporting would automate manual testing for me.
To set up an IntelliJ plugin project for Remote Robot, include the following dependencies in your <span class="pink" >build.gradle</span>:
Once your project has these dependencies, define test environment properties, such as the port:
Now, create an instance of Remote Robot, specifying the URL and port number:
Now, I can interact with any visible component on IntelliJ's screen, with actions like mouse clicks based on component coordinates.
Note that you don’t need to work with coordinates, the remote robot does it for you.
To obtain a list of visible components in IntelliJ, open your browser and navigate to localhost:8082 after running the Gradle task <span class="pink" >runIdeForUiTests</span> from your IntelliJ plugin project. This list displays all visible components rendered in the IntelliJ window.
Each component can be accessed through Fixture classes, which serve as access points for interacting with these components. Components are identified using XPath. It’s useful to have a unique XPath for major components you wish to interact with, like prominent buttons on your plugin's UI.
Here’s a snapshot of the components visible on an instance of IDEA.
This is what the idea window looked like when the above screenshot was taken.
The list of components are massive and this screenshot doesn’t cover all the components visible on the IntelliJ window.
You can fetch a Fixture class for a specific component using the <span class="teal" >find</span> method. Some fixtures, such as the main idea frame, should be organized into their own classes, extending <span class="pink" >CommonContainerFixture</span> to simplify interactions.
There are two major types of Fixture classes:
Here's an example of an <span class="pink" >IdeaFrame</span> class pointing to the main IntelliJ idea panel:
These methods enable interactions with IntelliJ's underlying APIs, such as the <span class="teal" >Project</span> API. For example, you can use them to determine if the IDE is in "dumb" mode, a critical aspect for some testing scenarios.
Dumb mode is a state where IDE doesn’t have all its features available, this happens when the project is still indexing. Once indexed, you’ll have all of the IDE’s features at your disposal.
Here's an example of how to use <span class="pink"> callJs</span> to check if the IDE is in "dumb" mode:
With a class representing the main IDE window, such as <span class="pink">IdeaFrame</span>, you can create tests to open specific files and perform actions. These tests can be executed using JUnit 5.
For instance, here's a simple test that opens a file and clicks the debug button:
This test waits for the IdeaFrame, ensures that the IDE is not in "dumb" mode, opens a specific file (UserController in this case), and clicks the debug button.
To interact with the debug button, a <span class="pink">ComponentFixture</span> is required. You can obtain it using a method like this:
To obtain the Xpath to identify the debug button, you can fetch it from the browser end-point:
Now, you're ready to run this test. Once executed, you will see the Remote Robot opening a specific file and clicking the debug button in your IntelliJ instance.
For better organization and to introduce pauses in your tests, Remote Robot provides two handy methods: <span class="pink">step</span> and <span class="pink">pause</span>.
Use <span class="pink">step</span> to organize sequences of code. For instance, you can split the code into two steps: opening a file and clicking the debug button.
To introduce pauses at specific points in your automation, the <span class="pink">pause</span> method is valuable. For example, to pause for one second:
With IntelliJ's Remote Robot, you're not limited to button clicks; you can script interactions with Java files and methods within your project.
This is a demonstration of our automation, in this case every java file gets visited and every testable method marked with a gutter icon gets directInvoked with autogenerated inputs.
Here’s what the test looks like at a high level :
<span class="pink">projectTreeView</span> gives us the list of all files seen on the project tree view.
This lists only what you can see on the IntelliJ screen.
Double clicking on an instance of <span class="pink">RemoteText</span> will open that file.
The gutter icons you see on the method line numbers - are the starting point of Unlogged’s user journey. I wanted a program to click on this icon and open the panel first.
Here is how to get a list of gutter icons.
In our case, I created a filter to click only on the “Process Running” icon.
Icons can be filtered based on many factors, here I’ve used the name of the image.
The line number where the gutter icon is rendered can be obtained using <span class="pink">GutterIcon</span> class.
With <span class="pink">TextEditorFixture</span>scrolling to any point in the editor but this is caret-based.
To scroll down to the correct point in the editor, the caret offset pointing to the start of the line that the gutter icon is rendered on is needed. This is possible with the help of a convenient method in the document model of the editor.
Once the editor has scrolled so that the icon is in view next is to
The Fixture for the “Execute Method” button is obtained the same way as seen with the debug button from the above example.
Once a method is executed, it may take sometime to respond depending on what it is doing. I added an artificial delay of 5 seconds so that the method completes its execution and the input/return values + CPU usage can be added to the report file.
Our Automation testing service is paired with a reporting service that records the inputs and return values of methods every time the methods are executed along with information like CPU and memory usage to monitor performance and visual/functional bugs.
I want to highlight 2 issues found using this automation.
1. Incorrect Rendering of Gutter Icons.
The correct gutter icon state was not rendered when multiple IntelliJ idea instances were open at once.
2. Performance Drops on longer sessions.
Longer running applications means longer sessions both in storage and duration.
The time taken to update a gutter icon post-execution and the average time taken for the same methods to display responses increased by as much as 40% when the sessions were longer.
More info and examples here :