Opening Insights: This article is structured in two main parts: the first delves into the nuances of product development, and the second outlines the architectural solutions and the envisioned future of the product. Given the comprehensive nature of our discussion, we encourage you to peruse the entire article for a thorough understanding.

In the following sections, I will unfold the narrative of our product’s evolution, tracing the path from early-stage prototyping to the polished end-product. The introduction of the GPT-3 API in the early months of 2023 heralded a shift towards enhancing the efficiency of job ad moderation on our platform, poslovi.infostud.com. Here, we manage the publication and moderation of hundreds of job advertisements each week, a process traditionally performed manually. Although meticulous, this approach is susceptible to inaccuracies and delays.

Confronted with many moderation guidelines, both explicit and implicit, alongside the transformative potential of generative AI models, we went beyond rigidly set KPIs and objectives. Our approach was characterized by an openness to experimentation, acknowledging the risk of failure while striving to maximize the likelihood of success through rapid prototyping. This method encompasses several key phases:

• Prototyping: Crafting high-fidelity prototypes that deliver practical solutions.
• Feedback: Gathering and interpreting feedback from users to inform development.
• Improvements: Refining the prototype in response to insights gained from user feedback.

While our aims were intentionally not defined as SMART goals, they were nonetheless focused and ambitious:

• Speed: Accelerating the moderation process for each job advertisement.
• Quality: Elevating the overall quality of job advertisements post-moderation.

This introduction sets the stage for a deeper exploration into the iterative development process and architectural innovation, underscoring our commitment to pushing the boundaries of what is possible with AI in the realm of job ad moderation.

V1 Prototype: Foundational Steps and Preliminary Concept Evaluation

The start of our first prototype- designated V1, was driven by the ambition to evaluate a foundational concept. The ultimate aim was to iteratively refine this concept into a robust final product. Our preliminary strategy aimed to equip administrators with insights into potentially contentious sections of job advertisements while concurrently extracting a diverse array of metadata from these ads. For the implementation of this prototype, we identified Streamlit as the optimal platform, given its flexibility and user-friendly interface.

The prototype was designed to find

• The existence of any form of discrimination within the advertisement’s text
• The inclusion of contact details such as email addresses, salary information, or phone numbers
• The presence of grammatical inaccuracies
• Whether the advertisement’s text inadvertently had multiple job listings

Included in the prototype’s functionality was the ability to extract

• The seniority level of the advertised position
• Relevant IT-related tags
• The geographical location of the job, pinpointing specific cities

The testing phase of the V1 prototype yielded encouraging results, demonstrating a high degree of accuracy in its operational performance. This initial success laid a solid foundation for later development stages, confirming the viability of our approach and the potential for further refinement and expansion of the prototype’s capabilities.

V2 Prototype: Broadening Functionality and Elevating User Experience

Building upon the insights gained from the first prototype, we embarked on the development of the second iteration, prototype version 2 (V2), intending to significantly enhance both the content quality and the visual appeal of advertisements. This ambition was realized through focused improvements in the user interface (UI) and user experience (UX), ensuring a more intuitive and efficient interaction for administrators.

UI and UX Improvements

• Introduction of a feature enabling the live streaming of processed data into the application, without the need for manual text input.

Functional Enhancements

• Acknowledging the considerable time administrators dedicate to formatting HTML versions of job ads, V2 aimed to automate this process. By employing a set of predefined rules, the prototype could transform ad text into HTML. This included the automatic identification of job descriptions, candidate profiles, benefits, etc. and their allocation to specified styles. For instance, the system was designed to recognize essential qualifications and skills from the ad text, correct any grammatical errors, remove discriminatory language, apply H2 headers in blue, and format benefits as bullet points.
• To address issues such as grammatical mistakes and discriminatory language in both Serbian and English, we broadened the prototype’s scope of functionality.
• A concerted effort was made to refine the ETL (Extract, Transform, Load) process, acknowledging the varied formats in which employers post job advertisements on poslovi.infostud.com, including PDF, DOCX, Images, and others. We intensified our efforts in processing and converting these formats into text in near real-time, after sending the text for further processing. The next sections will delve deeper into how this project catalyzed further investment in text processing infrastructure development.

This phase of development is not only aimed at enhancing the practical aspects of the platform but also at enriching the overall experience for both administrators and end-users, setting a precedent for continuous improvement and innovation.

V2.1 Prototype: Refinement and Advanced Change Detection

In response to the administrative challenge of finding textual modifications in job advertisements through mere visual inspection, we introduced an innovative feature in version 2.1 aimed at enabling precise monitoring of changes. This was achieved through the integration of a diff algorithm, setting up a robust foundation, and setting clear expectations for the final, production-ready iteration of our product.

In the evaluation of version 2, we decided to stop the automatic formatting of job ads for two main reasons:

• Employer Branding Integrity: Automatic formatting could alter the original layout provided by employers, affecting their brand presentation. By dropping it, we ensure the employer’s intended formatting is preserved.
• Resource Efficiency: The benefits of automatic formatting did not justify the significant resources needed for its maintenance and improvement.

Nonetheless, we kept and further developed all other functionalities from version 2.1 for inclusion in version 3, poised to be the definitive version of our product.

The enhanced product helps the direct annotation of sections of text within the administrative editor that may present issues. Administrators are empowered with the capability to either approve or dismiss suggested modifications. Internally, we affectionately refer to this feature as “Grammarly on steroids 😊,” a nod to its enhanced content editing prowess tailored to our specific application.

Figure 1 – Implemented solution

As we close the first chapter of our adventure into the world of AI-powered job ad moderation at Inspira Group, we find ourselves on the brink of a deeper exploration. Our focus shifts to the core of our innovation journey: data as our primary product. This chapter has shed light on our progress in improving moderation processes and hints at a major shift in direction.

Read the continuation of our story here.

Author:

Srđan Mijušković, Senior Product Manager at Inspira Group