Pablo Galiana, professor of Artificial Intelligence at the Faculty of Engineering, gave a webinar on what is behind the GPT Chat, to understand how the model is built and the challenges it presents.
Chat GPT is based on a generative model. Generative models are a field of study within machine learning that focuses on creating systems capable of generating data, such as text, images, or music. The history of generative models dates back several decades, with the first attempts in the 1960s using probabilistic techniques and language models. Over time, more sophisticated models were developed, such as generative adversarial networks (GANs) and LTSM models, among others. Since then, generative models have continued to evolve with techniques like Transformers. Chat GPT, which stands for "pre-trained generative transformer," emerged from these technological advancements.
A Transformer is a neural network architecture that was first proposed in 2017. This architecture has proven highly successful in natural language processing tasks and other machine learning applications. Over time, it was discovered that this neural network model could generate video, text, or images with specific characteristics.
“It feels like you’re talking to someone intelligent, when in reality what the model does is generate a text with a certain probability,” the specialist shared. This level of creativity is what the experts at OpenAI, the company that invented Chat GPT, call temperature. Depending on the prompt a person gives the chat, the response can be more or less creative.
ChatGPT's training is based on supervised learning and pre-training followed by fine-tuning. In the pre-training stage, a large dataset of internet text is used to expose the model to a wide variety of information. Fine-tuning is then performed on a more specific dataset labeled by humans.
What has been observed is a difficulty in formulating a prompt in the chat and ensuring the response matches the desired outcome. To address this, specialists at OpenAI are offering a course on prompt engineering to teach users how to ask the chat more precise questions. Knowing how to ask the right questions is key to obtaining the most accurate result possible. To achieve this, it is recommended to write clear and specific instructions and give the model time to process them.
Much has already been said about the potential impact of GPT chat and artificial intelligence in general. Some perspectives are more apocalyptic than others. In this regard, Galiana offered an optimistic view: while every technological change disrupts the way we do things and generally generates noise or fear, it always brings many opportunities.
