Written by Keisuke Inoue
By now, you’ve likely heard about ChatGPT, “a chatbot technology developed by OpenAI that uses natural language processing to generate human-like conversation” (this description was written by Davinci, a GPT3 AI model from OpenAI). The release of ChatGPT took social media by storm, and for good reason—people were amazed by its ability to provide seemingly thoughtful responses to a wide range of questions. However, others have pointed out its response inaccuracy (even going so far as to describe it as “fluent BS”). Regardless, ChatGPT represents a bold leap forward in AI’s potential to drive chatbot technology across a wide range of useful applications, without the need for extensive manual intervention during dialog setup and implementation.
ChatGPT is based on GPT3.5, the successor to GPT3, an extremely popular AI language model that was released in 2020. GPT3 (which stands for Generative Pre-trained Transformer 3) is a Large Language Model (LLM)—which means it’s an expansive neural network that’s designed to process a large data set of text to learn language and generate responses based on various inputs effectively. ChatGPT is a “fine-tuned” version of GPT3.5 and has a relatively thin neural network layer over GPT3.5 to optimize contextual dialog output.
Although similar models exist (including those developed by industry heavyweights such as Google, Meta, and Microsoft) and the list of competitors keeps growing alongside their ability and level of sophistication, GPT3.5 seems particularly revolutionary due to its convincing ability to handle a wide array of text-based tasks—including answering questions, writing essays, summarizing long texts, translating languages, and so on.
PandoLogic’s AI chat leverages various effective conversational AI techniques but does not use ChatGPT (or an LLM) to generate responses (for a detailed description of how PandoLogic’s AI chat works, click here). But is ChatGPT the future of recruitment chatbots? In this article, I’ll examine the potential of ChatGPT technology to drive recruitment chatbots forward. Previously, I used PandoLogic’s AI chat for hiring data scientists for my own team. Here, I sat down to leverage this example and see how much ChatGPT knows about the NLP Data Scientist role at PandoLogic.
Chatting with ChatGPT
Let’s start with a simple question:
Great answer! PandoLogic also offers a robust candidate management dashboard, pandoSELECT, and access to PandoLogic’s AI chat but overall, it’s a good description. Now, let’s get more specific and ask about the NLP Data Scientist role at PandoLogic (Note: this role is no longer open):
Not bad. But wait…is ChatGPT just giving us a general answer for any NLP Data Scientist role? Let’s dig a bit deeper:
Sorry ChatGPT—in reality, you do need to be fluent in Python to become an NLP Data Scientist at PandoLogic. Maybe ChatGPT didn’t read the job description (after all, not even ChatGPT can read everything on the internet). Fair enough—let’s provide the full job description (clipped from the following screenshot to save space), and see what happens:
There you go—just like Davinci (and other GPT3 SAS offerings out there such as Jasper.ai), it can decipher long instructions to provide a sufficiently nuanced context. An interesting feature of ChatGPT is that it “remembers” context, so now I can ask a question like this:
Correct! Well done, ChatGPT! Unfortunately, its memory and ability are limited—which becomes apparent after it loses context during longer back-and-forth interactions.
So, what else is ChatGPT capable of? It can also generate interview questions for a given job description, which is something PandoLogic is experimenting with, using different neural language models.
These are all good questions and reflect the job description well. However, evaluating responses is the greater challenge. Let’s see if ChatGPT can help us with this:
At first glance, it may seem as if ChatGPT is poised to take over all facets of recruitment and render many of us obsolete when OpenAI makes this technology available to businesses. But the truth is that it’s not perfect or omniscient, as others (including StackOverflow) have pointed out. In fact, ChatGPT failed to answer many of the typical questions candidates may ask:
Other questions ChatGPT failed to answer include:
- What does a typical day as an NLP Data Scientist look like at PandoLogic?
- Where is PandoLogic’s headquarters?
- Who is the CEO of PandoLogic?
ChatGPT does represent an improvement over GPT3 model technology (e.g., Davinci)—with GPT3, we often saw questions answered incorrectly:
Still, as the earlier question about Python illustrated, ChatGPT can provide incorrect answers if the right context is not provided or lost during a conversation.
Job searching and interviews—and their outcomes—are serious business, so we need to safeguard against any risk of inserting misinformation, inappropriate evaluation criteria, or unwanted biases into the process, which makes using a ChatGPT/LLM a challenge. Another problem with using a ChatGPT/LLM is that it requires more computational power, which may make chatbots’ response times slower and their operation more costly.
That said, operational bottlenecks often come from manual processes as well, and more automation and simpler architecture (when designed correctly) often yield more efficient systems.
The following are high-level overviews of a few ideas that PandoLogic is currently experimenting with:
1. Hybrid Approach for Candidate Screening
As discussed, there’s a set of questions that candidates often ask during job interviews. These questions relate to either the company, the job, or the process. PandoLogic’s AI chat is a whiz at leveraging key pieces of information that are aggregated and stored in PandoLogic’s database and using them effectively in her responses, to help make the onboarding process as seamless and informative as possible. This process currently relies on a classic intent-detection model that was developed in-house and can handle a limited set of questions.
If we switch to the ChatGPT/LLM model, in theory, PandoLogic’s AI chat will be able to handle some of the unforeseen questions as well. However, relying solely on this approach introduces challenges like reliability, cost, and response time. Therefore, the current best alternative may be implementing a hybrid version of these two approaches, where the current conversational AI/database model is used primarily and an LLM is used when intent detection output returns a low confidence value.
2. Generating Suggestions for Interview Questions
Currently, our automatic chat generation process is assisted by an NLP parser, which analyzes incoming job descriptions and extracts key data, including required skills and qualifications. Interview questions are defined based on available data and client requests. Current generation LLMs, including GPT3 and open-source rivals GPT-NeoX and BLOOM, can not only extract key skill requirements from job descriptions but can also form interview-format questions from them. Although these suggestions should be approved by human experts to ensure that the model isn’t inserting irrelevant or inappropriate questions, just having a list of suggested questions ready to insert into the interview design can be an improvement in efficiency over existing processes.
ChatGPT amazed us with its ability to generate sentences that exhibit language fluency and basic contextual understanding. However, it is not ready for the recruitment spotlight in its present form as demonstrated by some of the interactions above. Unless the proper context is provided, ChatGPT may help spread false information, perpetuate biases, and is susceptible to providing incorrect answers. At PandoLogic, we know it’s possible to establish a fair, honest, and objective recruitment process with technology and our conversational AI team seeks to solve these concerns while leveraging the latest in AI technologies.
As new advancements in AI capture public interest, it’s our mission to test, research, and then refine our own technology to ensure we have the most up-to-date offering available. We’re excited by the possibilities ChatGPT offers as we move towards a bold new future in recruitment.