AI Agent Test Automation Engineer (Python)
About the Team We are Quadcode, a fintech company excelling in financial brokerage activities and delivering advanced financial products to our global clientele. Our flagship product, an internal trading platform, is offered as a Software-as-a-Service (SaaS) solution to other brokers. We’re looking for a confident QA Automation Engineer (Python) to join our AI team and focus on testing developer interfaces, applied AI solutions, and complex multi-agent scenarios. You’ll play a key role in ensuring the quality and reliability of our AI agents and developer-facing products in real-world business applications. What We’re Building We’re creating the IDE of the future — a smart development environment powered by cutting-edge LLM agents. It goes beyond just code: our platform helps developers generate images, sound, video, design, and interfaces. This is an AI startup within a stable product company — the perfect combination of innovation and support. We’ve already built an MVP, have real internal users, and a team of skilled engineers. We're currently in internal testing and getting ready for production launch. We operate in GMT+3, with daily team standups at 12:00 GMT+3. Join us to build the next generation of software tools together! Core Responsibilities
- Develop automated tests in Python for complex scenarios.
- Build mock services and test environments for agent testing.
- Integrate tests into AI-specific CI/CD pipelines.
- Monitor AI system performance in production.
- Design quality metrics and dashboards for AI monitoring.
- Document test scenarios and establish best practices.
Developer Interfaces Testing
- Test SDKs, APIs, and libraries used by developers.
- Validate API documentation and code samples.
- Build automation for dev tools and CLI utilities.
- Test integration workflows and compatibility with various languages/frameworks.
- Test IDE plugins and extensions.
- Validate data schemas and OpenAPI specifications.
Applied AI Testing
- Test real-world AI use cases and production scenarios.
- Validate model outputs on production data.
- Test AI data processing pipelines.
- Benchmark AI inference performance.
Agent Scenario Testing
- Design tests for multi-agent systems.
- Validate interactions between AI agents.
- Test task chains and agent workflows.
- Ensure behaviour constraints and safety.
- Test integration with external systems.
Requirements
- 3+ years in software testing.
- Strong Python knowledge: pytest, unittest, mock, asyncio.
- Experience testing UI, APIs and SDKs.
- Understanding of ML/AI agent concepts.
- Experience with REST/GraphQL APIs, WebSockets.
- Familiar with performance testing tools (e.g. locust, k6).
- Experience with Docker and containerization.
- Familiarity with CI/CD and DevOps practices.
- Git proficiency.
- English level: B1 or higher.
- Fluent in Russian (C2).
Nice to Have
- Understanding of multi-agent system architecture.
- Experience with agent frameworks (LangChain, AutoGen, CrewAI).
- Working with LLM APIs (OpenAI, Anthropic, local models).
- Knowledge of prompt engineering and LLM testing.
- Experience with vector databases (Pinecone, Weaviate, ChromaDB).
- Familiarity with async/event-driven testing.
Bonus Points
- Python development experience beyond testing.
- Familiarity with ML frameworks (TensorFlow, PyTorch, Hugging Face).
- Experience with cloud AI platforms (AWS Bedrock, Azure OpenAI).
- Understanding of RAG architectures.
What We Offer
- Full-time remote work (External Vendor contract).
- Competitive compensation (EUR, USD, or crypto).
- Flexible working hours.
- Influence over the architecture of AI testing systems.
- Access to cutting-edge AI tools and platforms.
- A team of professionals specializing in AI.
- Positive and friendly atmosphere.
With a team of 500+ creative and tech-driven people across the globe, Quadcode is the place where ideas turn into impact, where every project is a chance to grow, innovate, and make your mark. Join us today, and let's shape the future of fintech together! Note: All applications will be treated with strict confidence. We thank all applicants for their interest, however, only those candidates selected for interviews will be contacted. #LI-JM1 Apply tot his job Apply To this Job