Why procedural text generation is the best?

Here's an article from LIFE magazine from March 3 1961 about a text generator that does some form of beatnik poetry and it is almost as good as a real thing. At least for the disinterested and uninitiated. 

The tone of the piece is a mix of tongue-in-cheek amazement and goodhearted irony. But nevertheless it is really curious piece of history. 1961 was the very infancy of the Natural Language Processing - but even then there was this semi-mocking tone regarding what NLP is capable of.  And the poem is decent enough. 

I think Lew Welch would have been a fan of this thing - there is this smoky blur of similarity of his cadences and of the generated poem.

Anyway. As a writer, text generation is one of my areas of interest. It is a tool worth exploring. And not because it can do job for you. 

The poem itself is the representative of the procedural text generation variety. In essence, it is a cookie-cutter template-based way of producing text out of available elements without taking context or semantics into consideration. It is as blunt and unelegant as it gets. Usually, the results are clumsy and borderline demented. But only if you expect it to do text writing the right way. If not - it can be incredibly efficient at its job if you know how to use it.

I can go even further and claim that of all its types, procedural text generation is probably the greatest application of natural language processing ever.

Here's why - it is a simulacrum1 operating with a simulacrum2 based on a simulacrum3 to do simulacrum4.

Cue Inception BRAAAAAM.

Let's unpack this.

3 Procedural Text Generation is based on an assumption that grammar rules and vocabulary is all there is for language. Such vital parts of the language as onthological and aesthetic contexts or semantics are left out of the picture for sake of keeping things simple. In other words, PTG assumes that "map IS territory". 

1 Then there the generative infrastructure - it is more or less a Rube Goldberg machine. Eerily futile, blunt instrument. 

It is a structure marked by placeholders and there is an inventory out of which the result is constructed. In fact, this structure is the text, except it is not, but it shapes the result. 

There can also be some rules that affect the choice of words, but it is optional. It can run at random just fine.

2 Traditionally, the formation of text is bound by narrative. Its intention is to transmit some sort of information.

On the other hand, the intention of PTG is to generate a text entity, not narrative or evoke imagery. Thus, the central element of PTG mechanism is more or less doing snapshots of the indeterminacy.

From the existential standpoint, the choice is rooted in numerous factors. But what if we take out factors completely and leave it blank - what kind of choice it will be? It won't be because it is a completely different thing now. It just operates similarly. That's where procedural text generation magic kicks in.

4 Basically, what happens is - a bunch of stuff is being thrown together and by accident it generates sense to the reader because the reader bears the baggage of experience and context and is able to embalm a piece of text with meaning. The text itself remains empty. It was made to be made.

This is substantially different approach that explores other facets of perception and cognition. 

On the other hand, because the text in this set-up is not actually the text but the template and its inventory - it turns the creative process the inside out.  Instead of conveying meaning - the spotlight is on the framework and its potential to generate combinations. 

The sum of parts generates new entity that contains information shaped in a certain way that is not really saying anything but can be interpreted if necessary. 

As a result, procedural text generation allows us to peek at the strange parallel dimension where things look mostly the same but function differently. The whole method opens up different creative possibilities in terms of interaction with the text and constructing narratives.

How OpenAI sells GPT-2 as NLP killer app?

So, the thing we've been waiting for a better half of the year just happened - OpenAI had released a full unabridged version of their infamous Natural Language Processing model GPT-2. 

You know, the one known for being capable of generating texts that look and read just like the ones written by actual human beings. The one OpenAI deemed to be "too dangerous" for the public release. The "Pandora's Box" that, if misused, would probably flood the world with fake news and otherwise manipulative synthetic content. The one that would make regular copywriters and journalists utterly obsolete. The one that birthed AI Weirdness blog, after all. You know, that one. 

Well, it is back again and it is a good reason to talk about one of its lesser-known aspects. 

One of OpenAI's competitive advantages is that it is "capped-profit" limited partnership, which means that OpenAI doesn't have to generate profit per se, but their sponsors (in limited partnership - folks like Microsoft) can use OpenAI's research for their own purposes (products or solutions within products). At the same time, OpenAI's primary performance indicators are more or less spread of their technological solution and maintaining its notoriety. 

Let's talk about the latter. 

The thing you need to know about OpenAI GPT-2 is that its marketing campaign was pitch-perfect. This is how you market the technological product to the unwitting masses. This is how you present a value proposition to those in the know. 

The secret is that you mostly don't really talk about the product itself but instead concentrate on its impending impact, the way it will shake things up, "make a difference," and so on. Hence "too dangerous" narrative. 

This approach creates a mild case of hysteria perpetuated by an eerie mystery, which is mostly based on a bunch of "um"'s and "uh"'s and whole lotta sweet technobabble. The goal is to exemplify what the thing is capable of subtly - leaving enough space for speculation. On the one hand, this approach created a stir in the media, thus guaranteed a higher degree of visibility than competing products. On the other hand, it signaled to the right people that there is a multi-faceted solution waiting for proper implementation.

Just look at the launch sequence and how it establishes the product. It kicked off with the press release that stated, "oh well, we guess we just made some kind of doomsday device. And we are really concerned about it. And we're definitely not going to do anything with it, because it would be ridiculous to put it out in the open just like that. So we are just going to keep under wraps for now". 

What happened next is the tortsunadomi of news pieces that retold the story over and over again and also added a few meandering thoughts full of piss and vinegar to the conversation. It is then quickly incorporated into the "fake news" and "deep fake" narratives and firmly contextualized as something definitely "threatening." As a result, the burning mystery is perpetuated and elaborated. 

The next step was to provide a proper showcase of the model capabilities while avoiding spoiling the big thing. It is a great sleight of hand. The whole thing is out in the open, but you claim it isn't. The showcase is made by an abridged version of the product presented as a technical paper and a tool for researchers. 

It played out perfectly - the presentation of expanded and diversified by multiple points of view on the model. As a result, - more opportunities became apparent.

There were a couple of NLP projects that provided GPT-2 a lot of publicity. For example, Talk to Transformer presented the original use case of GPT-2 - text prediction. The way it was presented neutralized any emerging criticism regarding its clumsiness (for example, it couldn't predict Austin 3:16). It was simply freestyling a bunch of stuff - sometimes it was more coherent, sometimes it wasn't. 

Then there was the Giant Language Testing Room that visualized the text analytics and showed the mechanics behind computer-generated and human-generated texts. And then there was Allen Institute's Grover than was designed to expose deepfake texts and also showcase how easily the model can recreate the text with a couple of settings tweaked (except when you ask it to write an essay on Tawny Kitaen for some reason). 

Also, there were numerous blogs that applied GPT-2 for creative purposes, like writing unicorn fiction or generating incongruous names. Blogs like AI Weirdness that basically celebrated model's bumbling nature.

In a way, this kind of presentation operates on a similar framework to Marvel Comics' built-up the arrival of Galactus back in 1966. First, The Watcher shows up and warns about the coming "doom." And everybody go "ah, we've gotta do something!" Then you get a harbinger in the form of Silver Surfer, who wreaks havoc but at the same time continually reminds us that "the worst is yet to come." The heat is right around the corner now. 

And then we get the main event - Galactus, who was just another supervillain in the long line of "ultimate threats" just ten times bigger and because of that characterized as ten times more imposing and menacing and dangerous because authors of the comic said so. 

And that's what happened. After months of build-up and speculations, in early November, the big 40GB 1,5 billion parameters version of GPT-2 was released to the public. And it was exactly what it said on the tin from the very start, just ten times bigger. However, the momentum generated by the build-up made it seem less apparent. But there is a catch which makes things even more interesting.

The thing is that "ten times bigger" doesn't translate into "ten times better" because the technology doesn't really work that way. What matters is how it manages to accomplish specific use cases and how does it correlate with the solutions for actual practical needs. 

GPT-2 is presented as a tool that generates texts that seem natural as if an actual human being wrote it. It is cool, but you need to understand that the model operates on the finite dataset, and all it really does is derives the text out of probabilities and context approximations. Text generation is not really generation. It is more of regurgitation in sophisticated combinations. GPT-2 output looks and reads just like a real thing in a way that the real thing can be bland and passable and not really saying anything, just like loads and loads of content on the internet, especially, its corporate parts. 

What GPT-2 really does is creating a serviceable text product that serves a specific well-defined function. But you can't really market that because it is not really unappealing beyond those in the know (say hi to BERT). However, that's the absolute peak of practical NLP. 

A perfect example is a conversational interface. You say "hi," and it says, "hello, how are you?" because that how conversations usually go. The other example is a verbal interpretation of data like stats in Mailchimp where you can get reports like "22% opened your newsletter, 5% of which clicked on the link, and 1% gave it a thumbs up", but it is beyond current GPT-2 use case. 

That's why the "fake news" narrative is so beneficial for GPT-2. 

On the surface, fake news seems to be a natural avenue for that kind of thing. That game is all about quantity, not quality, and the NLP generator can help with that. It was hammered over and over again in the press releases and subsequent news pieces. And the use case seems legit. Technically, the goal is to generate derivative content that will reinforce and possibly perpetuate prerequisite beliefs of the target audience. Just what GPT-2 is good at.

Except, that's not really the message of the narrative. The real message all along was, "this thing can be used like that if YOU won't use it." And that leaves an intriguing mystery of what is going to happen next, which in turn engages the target audience more than the product itself.  

MFT: OpenAI Talk to Transformer

You know it seems like a floodgate of different text generating projects had opened recently. You get a new one almost each week and each time things get a little bit more interesting. 

Previously on this blog, i have covered Google's Talk to Books and Poem Portraits. Both were promising but unremarkable thingies.

Now it is turn of OpenAI-based endevour.

The tool is called "Talk to Transformer". It is a natural language processing machine learning neural network designed to perform an autocomplete on the text based on an input query. Its modus operandi is simple - analyze an input text and predicts the most fitting completion sequence.

Long story short: it is interesting but all over the place and not exactly any good (but i like it, because it's only rock and roll).

Talk to Transformer (bizarrely enough still not abbreviated into Triple T, that's a serious oversight) was developed by Adam King. The application is based on OpenAI machine learning model GTP-2 that is known for its capability to produce coherent chunks of text that are barely distinguishable from the human-written text output. At least in the case of generic, bland-o, template-o, SEO wet dream text. 

So - GTP-2 is really intense when it comes to generating text. But the thing with generating text is that you need to provide a direction for the generative algorithm in order to get results. Talk to Transformer is doing just that. 

The idea behind Triple T is cute - to make an easier interface of showcasing the GTP-2 framework in action. It is not as much a tool itself but a glorified interactive advert. It is designed to inspire to use GTP-2. Which is great. 

The autocomplete feature is a good way of showing the flexibility of the system. It provides a reasonable amount of interactivity for the users and shows just enough to make a positive impression.

If you give the algorithm one word - it is going to built a text out of it. At all cost. The length of the text depends on the generative potential of the word from the corpus point of view. In addition to that, there are vector equations that add to the picture. 

The usual result is a word salad snipped from here and there into a text that is readable but not exactly informative but otherwise borderline entertaining. The reason for that is the limited amount of input. Due to lack of information the algorithm starts to generate text from the text it generates - the phenomenon known as recursion. The contexts mix-up and the whole thing just strolls wherever the eyes are looking at the moment. Because of that you get that sweet subtle nonsense. It is fun to play.

Things get better when you give the algorithm a complete sentence. With more stuff to play with - the algorithm is capable of producing a coherent text. If the sentence's message is straightforward - you get a couple of vignettes around the central message that bring some colors to the context. If the sentence is ambiguous or heavily relies on tongue-in-cheek meaning displacement - hillarity ensues. 

I've typed "Kylie Minogue fans don't masturbate" (known as one of the ways to decipher KMFDM band name) and got some saccharine stream of consciousness heavily reminiscent of teenage op-ed from the good old Tumblr. It confirmed the assumption that Kylie Minogue fans don't masturbate and then claimed that when they do, they listen to Taylor Swift's Shake it off (fair statement, tbh). Then the conversation switched to porn in general with references to Wall Street Journal study and then it was cut off mid-sentence on "This is why". 

This kind of structure is called "divagation" - when the subject of the conversation strays away to something somewhat related or even something else entirely. It's nice to know that NLP generative algorithms are capable of doing that, because jumping from subject to subject within one text was never my strong suit and it is probably good for SEO.   

If you give a significant chunk of text - things get weirdly scattershot. At first, i've dropped a couple of generic random generated sentences and got nothing really special. It was more of the same that managed to blend into generic text. Then i've tried more sophisticated writing and the algorithm more or less adapted itself to the while. It added nothing to the text, but was serviceable enough to make a pass. 

Then i thought it would be a good idea to try to autocomplete some short poetry. My usual suspects are well-known. 

- Ezra Pound' In a Station of the Metro delivered the best bad result - "Huh? I don't get really get it. This is too dense." Good to know that NLP algorithm can't dig ye olde Ezra.

- Kenneth Patchen's "The Murder of Two Men by a Young Kid Wearing Lemon-colored Gloves" fared better. The autocomplete was nice and simple "Now I Know You Love Me."

- William Carlos Williams' This is just to say. I've fed the whole text and the resulting autocomplete was "!!! So, I guess I'll be staying here again next time... Thank you so much!!". Kinda cute. 

Finally, i've typed some mad lib a strings of random characters into it. The resulting text was an arcane something-something out of the depth of its big bad booty text corpus. Funnily enough, the latter option seems to be the best way of using Talk to Transformer. The resulting text was inhuman slab of characters that persisted on the comprehension but denied contemplation. Here's an example:

"click click click clicking clicking clicking clicking CLICK CLICK CLICK CLICK CLICK CLICK

Uncomment this line to show
Ear Beard **********

PlayAlertSound "00000000000"

DOO DA DA DA DA DA DA DA DA DA DE DIO DA DA DA

honk hum tinkle tisk tisk tsk tsk tsk p.p. p.p. p.p.
hokum hohum hmm hmph hmmm mmmm hmm mmmm mmmmm mmmmmm mmmm hm hm mmmmmm"

Now this is something i would like to see more.

Overall, I really like what Talk to Transformer has to offer. It is a very good tool for tongue-in-cheek inserts and inhuman machine texts.