By Mag. Rafael Crescenzi
Language models have become a very powerful tool, and as AI-based personal assistants are used by the general public, the productivity of human work could scale to levels unthinkable a few years ago.
It's been said countless times: our world is being shaped by technology. From the invention of the printing press to the arrival of the internet, each innovation has been a step in the evolution of humanity. But now, perhaps, we are witnessing an unprecedented technological disruption.
This revolution is built on a technology known as artificial neural networks , which, in much simpler forms, were already used by Gauss and adopted their current form in the mid-20th century. While their potential was known, it wasn't until a little over 10 years ago that they began to gain prominence when, enabled by increasing computing power and data availability, they were used to revolutionize the field of computer vision, to the point of developing systems more accurate than humans.
These networks were characterized not only by being much larger than those used up to that point, but also by having configurations that allowed them to process a greater diversity of information, such as images and text, without requiring the intervention of human experts in its preprocessing. In other words, in a way, they learned to perform that preprocessing on their own.
Halfway between that milestone and the present day, the paper “Attention is all you need” was published, presenting a new configuration of artificial neurons called a “transformer,” which gave rise to the wave of algorithms we now know with large language models . Among them is the GPT (Generative Pre-trained Transformer) family , developed by OpenAI, which grew from 117 million parameters in the first version in 2018 to 175 billion in the third version, released in 2020, which is the core of chatGPT.
These are nothing more than transformer-based neural networks trained to predict the next word in a text sequence. Hence the term "pre-trained" (the "P" in GPT): they weren't taught to understand the meaning of a sentence, detect the sentiment of the text, or perform any other specific task.
Why was it done this way? Because of the amount of data. If the model needs to find the next word in a text, all texts in the history of humankind are viable data for training it; whereas if it wanted to train it to detect—for example—feelings, only properly categorized texts would be useful (several orders of magnitude less).
Why do they work? We could speculate that language encodes human knowledge, and the act of accurately predicting another word implies some understanding of what came before; but it's truly difficult to grasp—at least for me—why these networks are so powerful. It's much easier, however, to understand why they're so useful. To do this, we must first point out that they aren't limited to predicting just one word, since each word they generate can be added to the original text to generate another, and then another, and so on.
Furthermore, a text can often be crafted to elicit a specific response. For example, if I write (or copy) a long article containing information that interests me, and add "In summary:" at the end and ask someone to continue, most people would write a summary of what came before. Well, a language model does the same thing, because it has been trained to continue texts in the same way a human would.
Thus, with well-crafted prompts, these networks can be made to exhibit far more complex behaviors than those taught. If I write "America was discovered by," it will complete Columbus. If I say, "This is a system of equations; the solution for x and y is," it will not only perform the mathematical work but also explain it in words. If I give it a title and subtitle, I might even get it to write an entire article.
Recently, recognizing its potential, Microsoft took the initiative and secured an agreement with OpenAI to use chatGPT in Bing; both in the sidebar of the results page and in a standalone chat, similar to the original application. For its part, Google quickly announced a similar product, called Bardo, based on Lambda, the model that became popular last year because an engineer thought it had consciousness.
Tensions are running high, and it would seem we're about to witness a war between tech giants for the web search market. But is that really the case?
Certainly, these kinds of tools would replace many of the searches we perform, but precisely in cases where we're not interested in finding a website, but rather a specific answer. In my opinion, products like chatGPT don't threaten Google's search market share, but rather the size of the market itself, with a superior substitute: the personal assistant . But a truly useful one, one that eclipses HAL 9000 or makes JARVIS feel insecure.
This is a promise that was already made to us, and we were disappointed: Siri is over 10 years old, and Alexa and Google Assistant are close behind. But where those products failed, chatGPT seems to shine. It not only understands our requests with few errors, not only provides good answers from the first try, but also understands the context, allowing for feedback and corrections. In other words, it enables real dialogue between human and machine, allowing for an interaction with technology that, if implemented well, could revolutionize our productivity.
And the timeline for reaching that reality seems to be accelerating. Just in the last few weeks, MetaAI released LLaMA, a model with capabilities comparable to GPT-3 but open source and significantly faster and smaller. Stanford scientists created Alpaca, a chatGPT clone derived from LLaMA with adjustments generated from the ChatGPT API itself, for less than $1000 (compared to the millions it cost to train the original models). And, of course, OpenAI released GPT-4, a superior version of its flagship model, which not only reads text but also understands images.
All these developments, along with the massive adoption of the tools through Bing and Google Search; the incorporation of virtual assistants in Office and Workspaces; the development of Github's Copilot and Copilot X; plus the plethora of startups that will try to position themselves in that market; not only foreshadow a significant impact on the productivity of those of us who use them, but also a dizzying acceleration in the creation and improvement of the tools themselves.
This note was written by human intelligence
I'd love to follow the current trend and end this piece by saying it was written by AI, but that's not the case. That doesn't mean it didn't have its assistance: I used chatGPT to suggest continuations, to overcome the writer's block that comes with being a novice writer, and to improve my grammar. I even asked it to generate the texts that I then fed into other generative networks that convert text to images (DALL-E 2 and Stable Diffusion 2.1) to create the illustrations that accompany this piece. Ultimately, I didn't use AI to replace my work, but to do it much faster and with higher quality.
Rafael Crescenzi holds a Master's degree in Data Science, is an actuary, an IT entrepreneur, a university professor, and a doctoral candidate in engineering. He works as a professor at the Master in Data Science from the Faculty of Engineering from the Universidad Austral within the subject Implementation Laboratory II.
#DataMining #AI #LanguageModels #Bing #Microsoft #OpenAI #chatGPT #Google #LaMDA #Bardo
