LMS Course Content Scraper Engineer (Python)

Key Responsibilities:

• LMS Integration Development: Develop and implement secure and efficient methods to access and extract course content from target LMS platforms: Canvas, D2L Brightspace, Blackboard, and Moodle.
• Data Extraction & Preprocessing: Design and build mechanisms to extract various content types (e.g., text, documents, multimedia links, interactive components, videos, assignments, discussions) from LMS courses. This includes handling different file formats and structures.
• API Utilization: Leverage official LMS APIs where available and appropriate for content access.
• Web Scraping Techniques: Employ advanced web scraping techniques for platforms or content types where direct API access is limited, ensuring resilience to UI changes.
• Authentication & Authorization: Implement secure authentication flows to access user-authorized course content within the LMS. Assume provision of a service account that the component can use for LMS access.
• Error Handling & Robustness: Develop comprehensive error handling, retry mechanisms, and logging to ensure the scraper is resilient to network issues, LMS changes, and data inconsistencies.
• Scalability & Performance: Design the scraping component to be scalable and performant, capable of handling numerous course extractions efficiently.
• Documentation: Create clear and thorough documentation for the scraping component, including technical specifications, API usage, and troubleshooting guides.
• Collaboration: Work closely with the core development team, particularly with the "Course Content Ingestion Layer" and "LLM Processing Engine" components, to ensure seamless data flow.

Required Skills & Experience:

• LMS Expertise: Proven experience working with and understanding the architecture and data structures of major LMS platforms (Canvas, D2L Brightspace, Blackboard, Moodle).
• Web Scraping Proficiency: Strong expertise in web scraping technologies and libraries (e.g., Python with BeautifulSoup, Scrapy, Playwright, Selenium) and an understanding of best practices for ethical and robust scraping.
• API Integration: Solid experience integrating with RESTful APIs, handling authentication (OAuth, API keys), and managing rate limits.
• Programming Languages: Strong proficiency in Python is highly preferred, given its extensive libraries for web scraping and data processing
• Data Handling: Experience with data parsing, cleaning, and transformation to normalize extracted content into a consistent format.
• Security Best Practices: Understanding of security considerations related to accessing sensitive data and implementing secure coding practices.
• Problem-Solving: Excellent analytical and problem-solving skills to navigate complex LMS structures and handle varying content representations.
• Version Control: Familiarity with Git and collaborative development workflows.

Challenges & Considerations. The successful candidate will need to address several key challenges:

• LMS Diversity: Each LMS has its own unique architecture, APIs (or lack thereof), and content storage mechanisms.
• Content Variability: Course content can be highly diverse in format and structure within and across LMS platforms, requiring flexible and adaptable extraction logic.
• Rate Limiting & Stability: Designing the scraper to respect LMS rate limits and be robust against potential changes in LMS UI or API versions.
• Permissioning: Ensuring the scraper operates strictly within the bounds of user permissions for course access.

Desired Outcomes. The successful implementation of this component will enable our rubric review tool to:

• Automatically ingest course content from Canvas, D2L Brightspace, Blackboard, and Moodle.
• Provide a seamless and integrated experience for course evaluation.
• Ensure a consistent and clean data feed for the LLM processing engine.

Why join to us:

• Work Anywhere: Embrace the freedom to work from anywhere in the world. Your office could be a beach, a cozy café, or wherever you feel most inspired
• Flexibility: Wave goodbye to the 9-to-5 grind. We believe in a flexible working schedule that fits your life.
• Sponsored Education: We're invested in your growth. Enjoy sponsored education and training, ranging up to 50%.
• Personal Development: We're not just about work; we're about your growth. Craft your personal development plan and watch your career soar.
• Regular Salary Reviews: Your hard work won't go unnoticed. We conduct regular salary reviews to ensure you're fairly rewarded
• Career Advancement: The sky's the limit! Move up the ladder based on your performance, and your career trajectory could surprise you
• Corporate Events: From team outings to memorable celebrations, we know how to have a good time together
• English Classes: Enhance your language skills and open doors to global opportunities with our sponsored English classes
• Health Matters: Your health is our priority. Get your annual flu shot on us
• Work Equipment: We provide top-notch tools. Receive a compensation of $600 for your work equipment
• Paid vacation, sick leaves