AQA Engineer with LLM Testing experience

The company is a pioneering AI-driven research company providing in-depth stock analysis globally in multiple languages. Our mission is to bridge the knowledge gap in the investment world, empowering investors with AI-powered insights. By integrating our advanced AI technology into brokerage platforms and financial institutions, we enable millions of investors to make informed investment decisions.

We are looking for a Mid-level Automation QA Engineer to join our team in Poland. The role will primarily focus on API testing (80%) with UI testing (20%), alongside testing LLM-powered AI solutions. You will help design, enhance, and maintain automation frameworks, ensuring efficient and scalable testing—especially for AI/LLM-driven features.

Key Responsibilities:

Automation Framework & Strategy

• Contribute to designing and improving automation testing strategies.
• Develop and maintain automation frameworks for API, UI, and LLM validation.
• Ensure scalability, maintainability, and reusability of automation code.
  
  

Test Development & Execution

• API Testing: Develop automated API tests in Python using the Requests library.
• UI Testing: Create and maintain UI tests with Playwright.
• LLM Testing: Design and execute automated test suites for **Large Language Model (LLM)**based features, including prompt validation, output accuracy, and bias/error detection.
• Integrate automated tests with Postman collections where applicable.
• Execute performance, regression, and functional tests.
  
  

Continuous Integration & Deployment

• Integrate automation tests into CI/CD pipelines (Jenkins preferred).
• Work with Docker and Kubernetes to manage and scale test environments.
  
  

Collaboration & Documentation

• Collaborate with developers, QA, and DevOps teams to improve the QA process.
• Write test plans, strategies, and design documents.
• Track issues using Jira and manage test cases (Xray optional).
• Analyze logs, investigate failures, and troubleshoot in staging and production.
  
  

Requirements:

• 5 years of experience in test automation or a related QA role.
• Strong programming skills in Python.
• Solid experience in API automation (Requests library) and UI automation (Playwright).
• Hands-on experience testing LLM-based applications (prompt engineering, output evaluation, hallucination/bias testing).
• Experience with CI/CD tools (Jenkins preferred).
• Experience with Docker & Kubernetes for test environments.
• Strong analytical, problem-solving, and debugging skills.
• Experience with Jira for tracking and reporting.
• Ability to work independently and proactively in a fast-paced environment.
  
  

Nice to Have:

• Experience with Kafka.
• Familiarity with Xray for test management.
• Knowledge of AWS cloud services.
• Experience with ML/AI model evaluation metrics.