Job Title: LLM Expert

Department: AI / Technology / Operations

Reports To: AI Lead / ML Manager / Product Manager

Employment Type: Full-time

Job Summary

The LLM Expert is responsible for the day-to-day application, support, and optimization of Large Language Model (LLM) solutions. This role focuses on prompt design, evaluation, troubleshooting, and safe deployment of LLM-powered workflows to support business operations, customer interactions, and internal tools.

Tier 1 LLM Experts handle standard use cases, initial triage, and operational improvements, escalating complex model or system issues to higher-tier experts.

Key Responsibilities

A. LLM Application & Prompting

• Design, test, and refine prompts for accuracy, consistency, and safety.
• Apply LLMs to use cases such as:
• Customer support automation
• Content drafting and summarization
• Data extraction and classification
• Maintain prompt libraries and usage guidelines.

B. LLM Operations & Support

• Monitor LLM outputs for quality, bias, and hallucinations.
• Perform first-level troubleshooting for LLM-related issues.
• Identify failure patterns and escalate to Tier 2 / Engineering.
• Support rollout of LLM features and workflows.

C. Evaluation & Quality Control

• Conduct manual and automated evaluations of LLM responses.
• Compare outputs against defined success criteria.
• Document issues, improvements, and performance trends.

D. Safety, Policy & Compliance

• Ensure outputs comply with:
• Company AI use policies
• Data privacy and confidentiality requirements
• Responsible AI standards
• Flag potential safety, ethical, or compliance risks.

E. Collaboration & Documentation

• Work with product, engineering, and operations teams.
• Translate business requirements into LLM-ready instructions.
• Create user guides, prompt documentation, and FAQs.

Qualifications

Education

• Master's Degree and above.
• Equivalent experience may be considered.

Experience

• 1–5 years of relevant industry experience in working with LLMs or AI tools.
• Hands-on experience using platforms such as OpenAI, Azure OpenAI, Anthropic, Google, or similar LLM providers.
• Experience with prompt engineering or AI-assisted workflows.

Technical Skills

• Strong understanding of:
• Prompt engineering principles
• LLM limitations (hallucinations, bias, context windows)
• Token usage and cost awareness
• Familiarity with:
• APIs (basic level)
• JSON / structured outputs
• Evaluation techniques

Soft Skills

• Analytical and detail-oriented
• Clear written communication
• Ability to explain AI outputs to non-technical users
• Strong judgment and accountability

Tools & Technologies

• LLM platforms (OpenAI, Azure OpenAI, etc.)
• Prompt management tools
• Internal knowledge bases and datasets
• Basic scripting or automation tools (optional)