Migrating Katalon Studio Tests to Playwright with Model Driven Engineering

Introduction

Migrating automated tests from one platform to another is a common challenge for software teams. In our recent project at Berger-Levrault, we tackled the migration of functional test cases from Katalon Studio to Playwright, aiming to minimize manual effort and maximize reliability. This post shares our semi-automated approach, key results, and lessons learned.

Background: Katalon vs. Playwright

Katalon Studio is a popular tool for web, mobile, API, and desktop testing, offering a graphical interface and keyword-driven automation [Ereiz et al., 2019]. Playwright, on the other hand, is a code-centric framework supporting multiple languages (TypeScript, JavaScript, Python, etc.) and provides powerful APIs for browser automation.

The main differences lie in their programming languages (Groovy/XML for Katalon, TypeScript for Playwright) and their frameworks. Migrating tests means translating both the language and the automation API.

Our Migration Approach: MDE and Ktl2Plw

We designed a semi-automated migration process using Model Driven Engineering (MDE) [Verhaeghe et al., 2021]. Our tool, Ktl2Plw, is implemented in Pharo 12 [Bergel et al., 2013], leveraging the Moose platform for model navigation [Anquetil et al., 2020] and Famix for metamodel traits [Tichelaar et al., 2000]. The process follows a “horseshoe” migration strategy [Kazman et al., 1998], which consists of three main technical steps:

1. Cleaning

Katalon projects often contain inconsistencies that complicate automated migration. The cleaning step duplicates and refactors the source folder, renaming folders to remove spaces (which are invalid in Playwright identifiers) and converting Katalon-specific comments into proper Groovy comments. This ensures that subsequent parsing and transformation steps work reliably.

2. Import (Parsing and Symbolic Resolution)

The import step is divided into parsing and symbolic resolution:

• Parsing: All files in the Katalon project are visited. Groovy files are parsed using Tree-Sitter, a fast incremental parsing library. We developed an open-source Foreign Function Interface (FFI) in Pharo to interact with Tree-Sitter, enabling us to parse Groovy code and extract its structure. XML-like files (such as object repositories) are parsed using Pharo’s built-in XMLDocument package.
• Symbolic Resolution: After parsing, the tool builds relationships between entities. For example, a test step referencing another test case via the findTestCase keyword is resolved to a model link. This step ensures that all references and dependencies are correctly mapped in the metamodel.

3. Export (Code Generation)

Metamodel Design and Challenges

Our metamodel was constructed by analyzing Katalon documentation [Katalon Docs] and research papers [Ereiz et al., 2019; Octavia et al., 2021]. It currently includes 33 concepts, with 13 directly linked to Groovy entities. The metamodel is hybrid, combining Katalon-specific concepts (test cases, objects, keywords) with Groovy code structures (conditionals, try-catch, etc.). This hybrid approach was necessary because Katalon tests often embed Groovy logic, but it also increased complexity. In retrospect, separating Katalon and Groovy metamodels would improve maintainability and extensibility.

Example: Migrating a Test Case

The migration process translates Katalon test cases and keywords into Playwright code. For example, the Katalon openBrowser keyword is mapped to a Playwright navigation step using our KeywordLib. This library acts as a compatibility layer, allowing each migrated test step to call a familiar function that internally invokes the correct Playwright API.

Technical Example:

• The tool parses a Katalon test case, identifies the openBrowser keyword, and generates a TypeScript test file that calls KeywordLib.openBrowser(url).
• In KeywordLib, openBrowser is implemented to use Playwright’s page.goto(url) method, ensuring the intended browser action is performed.

Collaboration and Validation

Migration involved three teams: Katalon testers, Pharo engineers (tool developers), and Playwright engineers (validators). Each Katalon project was migrated independently of its original execution environment, so re-executing the original tests was not feasible. Our primary goal was to deliver syntactically correct TypeScript Playwright code.

Validation Workflow:

• After migration, Playwright engineers opened each generated test file in VSCode and used the Playwright linter to identify syntax and environmental issues (precompilation problems).
• Issues were classified as syntax errors, unknown identifiers, or missing properties. If a problem was recurrent, it was escalated to the Pharo engineers to improve the migration tool or KeywordLib. Otherwise, Playwright engineers fixed issues manually.
• The migrated project was then sent to the original testing team, who connected the new tests to their execution environments. Any remaining issues were either corrected by the testers or reported back for further tool improvement.

Results: Time Savings and Issues

We migrated 13 Katalon projects, totaling 1412 test cases. The average migration time per project was 16.8 seconds (excluding cleaning). Manual migration was estimated at 6 hours per test case, while our approach reduced total migration time by nearly 78%.

How Time Savings Were Calculated:

• Manual migration: 6 hours per test case × 1412 test cases = 8472 hours, plus 40 hours to design and code KeywordLib.
• Semi-automated migration: 218,920 ms total migration time (about 61 hours), plus 1.3 hours of manual checking per test case, and 40 hours for KeywordLib.
• Total semi-automated time: 40 + 61 + (1.3 × 1412) ≈ 1876 hours.
• Result: ~78% reduction in migration time.

Common Issues:

• Syntax errors (missing commas, parentheses, etc.)
• Unknown identifiers (undeclared variables, missing imports)
• Missing properties (unimplemented or mismatched keywords in KeywordLib)

Project	Objects	Test Cases	Custom Keywords	Migration Time (ms)	Problems Before Manual Check	Problems After Manual Check
paReg	378	44	6	113276	1385	605
paPart	1239	116	0	9332	84	82
paUrb	410	39	0	4217	101	41
paAdm	136	13	0	5880	71	68
paEmp	5	1	0	4179	0	0
paRen	1649	118	0	24852	1056	365
sec	433	69	0	6763	219	24
csaKvP	785	408	0	13908	98	32
taBl	642	171	0	6897	297	240
subv	486	98	0	9630	101	41
fdw	829	160	10	10565	427	297
pruBas	512	109	1	3295	563	313
notiFica	441	66	0	6126	101	5
Avg				16840	346.5	145.9

Table 1: Migration results across projects

Discussion: Pitfalls and Semi-Automation

Designing the metamodel was challenging, especially when mixing Katalon and Groovy concepts. We adopted an incremental approach, growing the metamodel with each migration iteration. However, importing test cases revealed non-trivial Groovy code structures (conditionals, switch statements, try-catch blocks) embedded in Katalon tests. This required a hybrid metamodel, increasing complexity and maintenance effort. In hindsight, separating Katalon and Groovy metamodels would improve maintainability and allow reuse in other migration scenarios.

Issue Type	Before Manual Check	After Manual Check
Syntax	16	0
Unknown Identifier	44	40
Missing Property	41	1
Total	101	41

Table 2: Remaining issues in subv project before and after manual checking

Future Work: LLMs and Beyond

Migrating from Katalon/Groovy to Playwright/TypeScript with static rules is tough due to syntax and API nuances. Large Language Models (LLMs) could help automate the migration of complex code structures, further reducing manual effort and improving accuracy [Eniser et al., 2024; Zhang et al., 2023; Yeh et al., 2024]. Future work will explore combining MDE and LLMs to target hard-to-migrate code, such as custom Groovy logic or complex authentication flows, and to further automate the resolution of unknown identifiers and missing properties.

Related Work

No prior work directly addresses Katalon-to-Playwright migration with MDE. Related research includes model-driven migration of JUnit to MSUnit [Jovanovikj et al., 2018] and semi-automated AST-based migration of JUnit to Gtest [Gao et al., 2024]. Our approach differs by focusing on keyword-object mapping and leveraging MDE for functional UI tests.

Conclusion

Model Driven Engineering proved effective for test migration. Pharo’s FFI enabled integration with external parsers, and our semi-automated process saved significant time. Future improvements include separating metamodels and leveraging LLMs for hard-to-migrate code.

References

• [1] Z. Ereiz, Automating web application testing using katalon studio, Zbornik radova Međunarodne naučne konferencije o digitalnoj ekonomiji DIEC 2 (2019) 87–97.
• [2] B. Verhaeghe, N. Anquetil, A. Etien, S. Ducasse, A. Seriai, M. Derras, Gui visual aspect migration: a framework agnostic solution, Automated Software Engineering 28 (2021) 6.
• [3] A. Bergel, D. Cassou, S. Ducasse, J. Laval, Deep into Pharo, Square Bracket Associates, 2013. URL: https://inria.hal.science/hal-00858725.
• [4] N. Anquetil, A. Etien, M. H. Houekpetodji, B. Verhaeghe, S. Ducasse, C. Toullec, F. Djareddir,
  J. Sudich, M. Derras, Modular moose: A new generation software reverse engineering environment, arXiv preprint arXiv:2011.10975 (2020).
• [5] S. Tichelaar, S. Ducasse, S. Demeyer, Famix and xmi, in: Proceedings Seventh Working Conference on Reverse Engineering, 2000, p. 296–298. doi:10.1109/WCRE.2000.891485.
• [6] R. Kazman, S. G. Woods, S. J. Carrière, Requirements for integrating software architecture and reengineering models: Corum ii, in: Proceedings fifth working conference on reverse engineering (Cat. No. 98TB100261), IEEE, 1998, pp. 154–163.
• [7] L. Włodarski, B. Pereira, I. Povazan, J. Fabry, V. Zaytsev, Qualify first! a large scale modernisation report, in: 2019 IEEE 26th International Conference on Software Analysis, Evolution and Reengineering (SANER), IEEE, 2019, pp. 569–573.
• [8] S. Brown, Software architecture for developers, Coding the Architecture (2013).
• [9] R. P. Octavially, R. R. Riskiana, K. A. Laksitowening, D. Sulistyo, M. A. Kusumo, N. Selviandro,
  Test case analysis with keyword-driven testing approach using katalon studio tools (2022).
• [10] B. Verhaeghe, A. Hosry, N. Hlad, Evref-bl/pharo-tree-sitter: v1.0.2, 2025. URL: https://doi.org/10.5281/zenodo.15089054. doi:10.5281/zenodo.15089054.
• [11] H. F. Eniser, H. Zhang, C. David, M. Wang, M. Christakis, B. Paulsen, J. Dodds, D. Kroening,
  Towards translating real-world code with llms: A study of translating to rust, arXiv preprint
  arXiv:2405.11514 (2024).
• [12] Q. Zhang, C. Fang, Y. Xie, Y. Zhang, Y. Yang, W. Sun, S. Yu, Z. Chen, A survey on large language models for software engineering, arXiv preprint arXiv:2312.15223 (2023).
• [13] H.-W. Yeh, S.-P. Ma, Y. Chen, Test case migration from monolith to microservices using large language models, in: 2024 IEEE International Conference on e-Business Engineering (ICEBE), IEEE, 2024, pp. 29–35.
• [14] I. Jovanovikj, G. Engels, A. Anjorin, S. Sauer, Model-driven test case migration: The test case reengineering horseshoe model, in: International Conference on Advanced Information Systems Engineering, Springer, 2018, pp. 133–147.
• [15] Y. Gao, X. Hu, T. Xu, X. Xia, D. Lo, X. Yang, Mut: Human-in-the-loop unit test migration, in:
  Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, 2024, pp. 1–12.