Automated tests (CI) for PRs and merges

[mkaze]

I was wondering if there is a plan to setup continuous integration for running the unit and integration tests automatically on each pull request as well as on merges to master branch. I think as the project and the number of contributors grow, this would be a useful addition to get a quick feedback and keep the project stable. Of course, each contributor/reviewer can independently run the tests locally on their own machine before submitting a PR/merge; however it may not be efficient and is kind of error-prone.

[zaleslaw]

The current status of TC:

1. Internal JetBrains TC: api-test, example-test, maven-publication, build artifacts
2. Trigger on each commit to the master branch

Plans:

1. Run tests/examples on proposed PRs
2. Trigger on PR (maybe wrong way)

Alternatives for the future:

1. Make integration with a simple OK from the internal TC build server
2. Make public TC with credentials for top contributors (low probability in the nearest future)
3. Kind of GitHub actions integrations

[zaleslaw]

I discussed your issue with the team and we made the following solution for this moment

Plans: Run tests/examples on proposed PRs + Make integration with a simple OK from the internal TC build server is made and will be applied for the future PRs via TC Build features, thanks for the idea and the lighting of the pain for the contributors @mkaze

But there is no trigger at this moment, it will be run manually by request to me.

[mkaze]

But there is no trigger at this moment, it will be run manually by request to me.

@zaleslaw All right, thanks! That's much better than nothing. Hopefully, in future we will adopt a more public and automated process for this to increase the efficiency of development and feedback loop for contributors.

[zaleslaw]

At this moment the trigger on PRs commits or access for the users outside JetBrains org is prohibited due to security reasons (it's easy to make DDOS on our TC servers), hope to move forward with this question.

I'm not familiar with GithubActions, but will learn this possibility to do something

[avan1235]

I have tried to create some simple working example of such GH action as I've been working with GH a few times now but the tests were then failed with the unknown for me reason. You can see the definition of such workflow here but there is a problem visible in this run

[zaleslaw]

Great, @avan1235 I'll try your approach. So, I know the reason for this test failing. It's a flaky test due to some asserts are not true sometimes due to TensorFlow non-determinism. I need to revisit conditions in these tests related to the different optimizer usage and relax assert conditions there.