I don’t know why, but I’ve been thinking about this for a few months now, and decided to get to the bottom of it.

My usual go-to is Wikipedia for these matters, and there is indeed a relevant article titled Decipherment of Cuneiform. I was not satisfied by the amount of detail it provided, however. For example, it says: “Della Valle understood that the writing had to be read from left to right”. What? How?? I also looked up one or two of the linked books in Wikipedia and searched for YouTube videos, which were similarly unsatisfactory.

So how, specifically, did people learn to read cuneiform?

One thing I learned in the process is that there are many different types of cuneiform. This post only deals with Old Persian cuneiform, the first to be deciphered.

Necessary caveats: I am (obviously) not an expert. While I wrote the post in its entirety, I also relied on Claude Opus for discussion and translation of original sources in Latin, German, and French.

Awareness

Travelers visiting ancient Persian lands regularly encountered sculptures accompanied by mysterious ‘decorations’:

Many thought this was text, and tried to decipher it, without making much (known) progress. Nonetheless, there were some early guesses which turned out to be correct. For example, the Italian traveler Pietro Della Valle, who had been in the region between 1616-1621, saw the glyph

𐎠

and observed (translation by Claude):

What gives me an indication that it may be written from left to right in our manner is the character, which is composed of four similar pyramidal figures — three upright with the point downward, and one lying on its side above them. For of the pyramidal figures, the head of this writing, as one sees in all the characters, is the broad part, which always stands uppermost when they are upright. Now in that pyramidal figure lying above the three that stand on end, since its head — that is, the broad part — is on the left, and its tail — that is, the point — on the right, it shows that the beginning of the writing is from the left side toward the right. Still, I do not affirm it for certain.

Simply put, because the lower three ‘triangles’ of the example glyph get narrower as they go down, and the upper ‘triangle’ gets narrower as it goes from left to right, the text might be read left to right. But he’s not sure.

(This is all Della Valle had to say, if you were wondering if I left something out. Not strong reasoning IMO, but it is what it is.)

Not everyone agreed that the ‘decorations’ were actually text, however. Thomas Hyde, who coined the term cuneiform¹, apparently believed that the signs were meant to serve as ornaments. (Charitably, perhaps he saw inscriptions with just a few lines? I cannot imagine seeing literal walls of text like the one photographed above and concluding “Ah, decoration.”)

Such was the state by the 1760s - controversy on whether cuneiform even represented writing proper. This changed with Carsten Niebuhr.

Details Emerging

Carsten Niebuhr, a German mathematician and cartographer, excelled in his studies, and was thus recommended to be part of the Danish Arabia expedition at the age of 27. You can see his journey route below:

Image from deSyracuse.

This was a dangerous journey: Niebuhr was the only one of the six members to survive. That he survived is fortunate for us, because Niebuhr visited Persepolis, an ancient Persian city with many cuneiform inscriptions, and made a number of critical contributions, getting the decipherment ball rolling.

First, Niebuhr left highly accurate copies of the inscriptions at Persepolis, and published them in 1778, in a series of tables labeled from A to L. This allowed anyone with access to his book to contribute, not just people (like him) who were fortunate enough to see the inscriptions in person.

Second, Niebuhr noticed that there was not just one but three types of script. See an image of how he classified them below (on the right, tables B, C, and D):

Not having seen the inscriptions myself, it is difficult to know how easy it is to tell in person, but once they are organized like this it is relatively easy to see the differences. In type B, for example, the wedges only appear vertically or horizontally, except when the wedge is used alone. In type C, various forms of diagonally oriented wedges appear. Type D is somewhat similar to type B, but there are generally more wedges. Probably there was some physical separation between the blocks, making it easier for Niebuhr to tell that they were meant to be distinct.

Finally, Niebuhr noticed that one of the three scripts he identified was the simplest, making it the prime target for decipherment. Regarding type B above, Niebuhr says (translated):

The number of letters is therefore by no means as large as one might perhaps have supposed from the copies of my predecessors. I have collected the various letters of one alphabet, the one that occurs most often, and have found no more than 42 of them.

Niebuhr has, by this point, provided accurate records and established key basic facts about the script, including different script types and which would be easiest to decipher (the type in Table B, which we will alternatively call “Old Persian Cuneiform”). But he has not yet deciphered the meaning of the text.

Breakthrough

As Niebuhr’s records spread, Oluf Gerhard Tychsen guessed that the diagonal wedge (𐏐) was likely a word separator. It is unique, as discussed above, in that it is the only diagonal wedge to appear in the glyphs. Furthermore, separating by the wedge yields roughly word-sized chunks.

This observation in turn made it easy to realize that there is a common recurring sequence in the text. See if you can find it yourself (here, I quote part of the DNa inscription):

𐏐 𐎠𐎭𐎶 𐏐 𐎭𐎠𐎼𐎹𐎺𐎢𐏁 𐏐 𐎧𐏁𐎠𐎹𐎰𐎡𐎹 𐏐 𐎺/𐏀𐎼𐎣

𐏐 𐎧𐏁𐎠𐎹𐎰𐎡𐎹 𐏐 𐎧𐏁𐎠𐎹𐎰𐎡𐎹𐎠𐎴𐎠𐎶

𐏐 𐎧𐏁𐎠𐎹𐎰𐎡𐎹 𐏐 𐎭𐏃𐎹𐎢𐎴𐎠𐎶 𐏐 𐎻𐎡𐎿𐎱𐏀𐎴𐎠/𐎴𐎠𐎶

𐏐 𐎧𐏁𐎠𐎹𐎰𐎡𐎹 𐏐 𐎠𐏃𐎹𐎠𐎹𐎠 𐏐 𐎲𐎢𐎷𐎡/𐎹𐎠

This is parsed favorably, so you should be able to make it out: “𐎧𐏁𐎠𐎹𐎰𐎡𐎹” is a frequently recurring pattern. What might it be, though? A name? A title? An article like “the”? Tychsen made some guesses as to how to read the script, but it was not the most accurate.

It is at this point where Georg Grotefend, a 27-year-old scholar, made a series of bold informed guesses that led to significant progress.

In his 1802 submission to the Göttingen Society, Grotefend suggested the following:

The script is alphabetic

If we accept 𐏐 is a separator, the glyphs in Niebuhr’s Table B cannot represent individual words. Not only are there too few observable glyphs for that, it is implausible that there would be repetitions of exact phrases over and over. Thus, the script is likely alphabetic.

The language it describes is likely Old Persian

If the script is alphabetic, certain glyphs are likely vowels. We can identify which of them are vowels, by looking at glyph frequency, since vowels can be expected to occur more often. Indeed, three glyphs are notably more frequent: 𐎠, 𐎶², and 𐎡.

Grotefend argued the usage of these glyphs closely fits vowel usage in the Avestan language (the language used by the Persian peoples in 1500-400 BCE). Therefore, we can deduce that the language the glyphs are recording is similar in nature. This makes sense because the glyphs were found in Persian lands, and Avestan is a Persian language.³

The script is read left-to-right

Grotefend provides a stronger reason why the script is read left-to-right: namely, that patterns are better preserved when read in such a manner.

Here, Grotefend did not give a specific example, but it’s fairly easy to find one. Consider the often-recurring pattern “𐎧𐏁𐎠𐎹𐎰𐎡𐎹”, and how it is recorded in Niebuhr’s original table. We can see an instance of the left part of it appearing on the right side of second line and the right part at the left side of the third line. This provides reasonably strong indication that the glyphs are read left-to-right, top-to-bottom, just like the Latin alphabet usually is.

The pattern “𐎧𐏁𐎠𐎹𐎰𐎡𐎹” likely means “king”, with the king’s name appearing to its left

With the reading order in mind, Grotefend observes that Niebuhr’s Table B and G start as:

Table B:   𐎭𐎠𐎼𐎹𐎺𐎢𐏁𐏐𐎧𐏁𐎠𐎹𐎰𐎡𐎹

Table G: 𐎧𐏁𐎹𐎠𐎼𐏁𐎠𐏐𐎧𐏁𐎠𐎹𐎰𐎡𐎹

From this, he conjectures that “𐎧𐏁𐎠𐎹𐎰𐎡𐎹” is a title, most likely “king”, and that the sequence preceding “𐎧𐏁𐎠𐎹𐎰𐎡𐎹” is the king’s name. This makes sense as there are also inscriptions where the “king” sequence appears in succession: “𐎧𐏁𐎠𐎹𐎰𐎡𐎹𐏐𐎧𐏁𐎠𐎹𐎰𐎡𐎹𐎠𐎴𐎠𐎶”, which would mean “king of kings”.

Identification of the kings in the inscriptions: Darius and Xerxes

Tables B and G are in fact almost identical in structure:

Table B:

𐎭𐎠𐎼𐎹𐎺𐎢𐏁𐏐𐎧𐏁𐎠𐎹𐎰𐎡𐎹𐏐𐎺𐏀𐎼𐎣𐏐 // [name1] king [word1]

𐎧𐏁𐎠𐎹𐎰𐎡𐎹𐏐𐎧𐏁𐎠𐎹𐎰𐎡𐎹𐎠𐎴𐎠𐎶𐏐 // king of kings

𐎧𐏁𐎠𐎹𐎰𐎡𐎹𐏐𐎭𐏃𐎹𐎢𐎴𐎠𐎶𐏐 // king [word2]

𐎻𐎡𐏁𐎫𐎠𐎿𐎱𐏃𐎹𐎠𐏐 // [word3 with 𐏃 modifier]

𐎱𐎢𐏂𐏐𐏃𐎧𐎠𐎶𐎴𐎡𐏁𐎡𐎹𐏐 // [word4] [word5]

…

Table G:

𐎧𐏁𐎹𐎠𐎼𐏁𐎠𐏐𐎧𐏁𐎠𐎹𐎰𐎡𐎹𐏐𐎺𐏀𐎼𐎣𐏐 // [name2] king [word1]

𐎧𐏁𐎠𐎹𐎰𐎡𐎹𐏐𐎧𐏁𐎠𐎹𐎰𐎡𐎹𐎠𐎴𐎠𐎶𐏐 // king of kings

𐎭𐎠𐎼𐎹𐎺𐏃𐎢𐏁𐏐𐎧𐏁𐎠𐎹𐎰𐎡𐎹𐏃𐎹𐎠𐏐 // [name1 with 𐏃 modifier] king

𐎱𐎢𐏂𐏐𐏃𐎧𐎠𐎶𐎴𐎡𐏁𐎡𐎹𐏐 // [word4] [word5]

There is a clear pattern between the two; what can we do with this? Here, there is a difference between the interesting story on Wikipedia, and what I could find in Grotefend’s original text.

Wikipedia’s Story

Note that [name1] is mentioned in Table G. The story goes that it was already known that Persian kings described themselves in this formula. For example, here goes the translated description of a Persian king in the 200s (Shapur I):

… lord Shapur, king of kings of the Iranians and non Iranians… son of … lord Ardashir, king of kings of the Iranians…

Then the recurrence of [name1] in Table G likely means that [name2] was a son of [name1], with the 𐏃 modifier perhaps doing a grammatical job related to the “son of” meaning. With this in mind, we can make an interesting observation about Table B: there, [word3] does not have the king title following it, unlike Shapur’s description and Table G, which explicitly note the father is a king. Thus, we need to find a Persian king whose father was not a king, but whose son was.

Two Achaemenid kings fit that pattern: Cyrus II and Darius I.⁴ A distinguishing factor is that for Cyrus II, the name of his father and son were the same (Cambyses). However, [name2] and [word3] are clearly different. So [name1] must be Darius! From here, we can deduce the sound values of the cuneiform, since we earlier suspected it to be alphabetic, and eventually decipher it all.

Grotefend’s Description

Unfortunately, the tidy story is not what I could find in Grotefend’s 1802 submission.

Instead, Grotefend noted that [name1] appears in both Niebuhr’s tables B and G, and thus deduced that the kings first mentioned in B and G were father and son:

… from which I gathered that the kings praised in these inscriptions were father and son. Since therefore the name of Darius — which the sacred codex calls Darjavesch — seemed to square with that word which Tychsen read Malkéusch (Dârhéusch), and the name of Xerxes with that which Tychsen read Osch patscha (Khschhêrschê): what the royal title was could not long escape me.

Thus, Darius and Xerxes are Grotefend’s guesstimates of the names based on the father-son relationship and the reconstructed sounds of Tychsen, not from a genealogy search.

Be it genealogy or guesstimate, Grotefend has concluded that [name1] is Darius and [name2] is Xerxes. Having established that the glyphs are alphabetic in nature, we can start deducing what sounds the glyphs would represent.

Phonetic Deduction

Once a name is known, mapping it back to sounds might not seem too hard. But consider: would the ancient Persians have pronounced Darius the same way as we English speakers do? Probably not: “Darius” is in fact a Latin form of the Greek term for the King, so it is twice removed from the name in its native language. Thus, for accurate decipherment, one must try to reconstruct how the ancient Persians might have called their king.

Grotefend reconstructed Darius’s name from the Bible, where Darius is called “Darjavesch”. Without describing why, he slightly modified this to “Dárheúsch”,⁵ which was then used to identify the glyph sounds:

Conveniently, [name2] (Xerxes) shares all glyphs with Darius except its first. Guessting the first letter to be kh based on Xerxes’s name, we get

The “kh” read is also corroborated by our key word “king”, “𐎧𐏁𐎠𐎹𐎰𐎡𐎹”, which starts with the same glyph as Xerxes; Grotefend says that an earlier scholar reconstructed the Old Persian term for king to be “Khscheio”.⁷

Grotefend, not being an expert in ancient Persian himself, used a dictionary of Old Persian words to reconstruct even more. For example, 𐎺𐏀𐎼𐎣 would read as “e?r?” based on what we have deduced. Grotefend thinks this word is an adjective (as it appears after “king”), and finds from the dictionary the word “eghré”, meaning “strong” or “great”. Thus he further connected “𐏀” with “gh” and “𐎣” with “é”. Through such a process, he constructed the following table and provided the meaning of the shorter inscriptions from Niebuhr:

Well, then, this looks good, are we done now?

Not quite. When compared with a modern reconstruction, Grotefend’s phonetic reconstruction wasn’t even half correct. As mentioned, Grotefend relied on a dictionary of Old Persian words. However, later scholars pointed out that the dictionary he relied on had many problems itself, leading to incorrect deductions. That 𐎺𐏀𐎼𐎣 word earlier? Modern scholars would read it as “vazraka”, not “eghré”, so it was quite far off.

Without an accurate phonetic reconstruction, it was thus impossible to read text that did not follow patterns already established. This contributed to skepticism amongst the academic experts in the area; to them, using Grotefend’s table to decipher unknown text probably led to sounds that seemed meaningless or even impossible in the ancient Persian language.

Grotefend’s semantic reading was validated when hieroglyphs were deciphered. In 1823, an Egyptian vase was found to have both the cuneiform “𐎧𐏁𐎹𐎠𐎼𐏁𐎠𐏐” and hieroglyphs spelling out “Xerxes”. This further supported Grotefend’s guess that [name2] was in fact Xerxes. Nonetheless, this did not rescue the ultimately incorrect phonetics.

Expert At The Rescue

Enter Eugène Burnouf. Born to a scholarly family, Burnouf was an expert in ancient Iranian and Indian languages. He had deciphered Avestan (an ancient Persian language) script, and thus had a deep knowledge of the language, precisely what Grotefend lacked.

Here, the popular story seems to be that “Burnouf discovered that the first of the inscriptions published by Niebuhr contained a list of the satrapies of Darius”, which is… another interesting but inaccurate story.⁸

Having written this, though, I sympathize with simplifying, as it’s impossible to fully communicate what Burnouf did. From this point on, decipherment becomes less about exciting ideas and more about the diligent application of language-specific knowledge, making it difficult to distill into a single compelling observation. Nonetheless, here is my hopefully more instructive summary of Burnouf’s methodology based on my reading of Burnouf’s book.

The key method for him was to note that there were some words that were close but not exact matches to known geographies and gods. I built the following table to simulate the knowledge that he had in his mind and show the technique: try to match the Grotefend reading to the closest known word, knowing that the Grotefend reading is partially accurate.

If you tried matching the words, hopefully it wasn’t too hard to match “auroghda” with Ahura Mazda, as both start with the “au” vowels and end with “da”. If we accept this match, we can identify that the “m” sound we would expect from Ahura Mazda is completely missing from Grotefend’s reading. Thus, we can correct Grotefend’s reading of “𐎶” from “o” to “m”.

We can be surer of this reading, because it makes other readings closer to our matches. Consider the reading “asoatscho”, which was probably one of the harder to match. Its reading would now be “asmatschm”, close to “ashmanem”. This is again almost accurate, while suggesting that “𐎴” should be read as “n” instead of “tsch”.

To complete the matching process: reading “𐎴” as “n” completely reconstructs “-anam”, so we can again be more confident. “akhaotschoschoh” becomes “akhamnoschoh”, which matches “Achaemenid” sans the suffixes. Probably the easiest match was “bakhmrosch” matching Bactria, again sans suffixes. Again, this is a simplification of what Burnouf did - he had many more Persian words and cuneiform words to match consistently - but it illustrates the rough process through which he corrected Grotefend’s reconstruction.

Contrary to the popular narrative, it is only after this series of corrections was made that Burnouf recognized there was a list of satrapies in one of Niebuhr’s inscriptions - he did not start with the recognition that the list of satrapies was in the inscriptions.

Burnouf’s phonetic reconstruction was still not completely correct, but it was substantially closer than Grotefend’s. Over the years, later scholars could further refine the phonetic reconstruction, eventually leading to its modern form:

Epilogue: The Behistun Inscription

We have told the story of how the cuneiform type in Niebuhr’s Table B, which he identified as the easiest to decipher, was gradually understood. But what about the types in Tables C and D?

Here, the British officer Henry Rawlinson played a role similar to Niebuhr. During his time in Persia, Rawlinson provided an accurate depiction of the Behistun Inscription, which has inscriptions in all cuneiform types, but which are much longer. This seems to have been enough source material for the decipherment of the other types. Just look at how much is inscribed: all the “lines” you see on the left, on the right, and on the bottom of the photograph are all in fact cuneiform from what I can see.

But that’s a story for another day, and I’ve satisfied my itch for a while. When (if?) we come back to this topic, we’ll describe what ideas were used in deciphering the other, substantially more complex cuneiform types.

Footnotes

In the last few years, however, we have made replicas of our thought process in the form of large language models (LLMs). LLMs are quite good at thinking - not only do they have a superhuman breadth of knowledge, they can reason well enough to solve long-standing math problems.

This has made me wonder. If LLMs can think, and often reach even better conclusions than I can, why think? Will future generations still need to think?

Think, to interact in real-time

One thing current LLMs struggle to do is to react in real-time. In many social situations, however, being able to answer immediately while on one’s feet is a critical requirement. Thus, one may argue that thinking is essential to navigate these cases.

I am doubtful. LLM inference is being optimized every day. The capabilities to “read” social situations will also grow rapidly if that ever starts to become an economic bottleneck for AI in doing work. Thus, even if this is a reason one should think right now, it is not a durable reason to think.

Think, to gain status

Let’s start by looking at something else: physical strength. Historically, physical strength was necessary to do many jobs: farming, blacksmithing, sailing, et cetera.

At some point, physical strength became non-essential for many valuable jobs. The likely pivotal point was the industrial revolution, as we started using powerful machines to do the physical work, instead of doing it ourselves.

As a result, physical strength is primarily a status symbol in everyday life now, not a functional barrier in doing work. Being better at sports or weightlifting has little practical value, but it increases one’s esteem.

LLMs are automating thinking, just like the industrial revolution automated physically demanding tasks. Under this analogy, being capable of good thinking would still be valuable as a status symbol, rather than being an economically valuable capability. Good thinkers would be admired the way we admire people who can lift heavy objects, with the simultaneous understanding that machines are in an incomparable, separate category.

Think, to be healthy

Let’s extend the analogy with physical strength. Exercise is beneficial for one’s health, in addition to status. As a result, health is a big reason we continue to engage in physical activity, even when physical strength has little direct economic value.

Just like a prolonged period of not exercising can have harmful effects on the body, prolonged periods of not thinking can have harmful effects on one’s mind and health. For a direct example, without a regular habit of thinking, one may grow indulgent and intolerant of inconveniences, in a way that is self-harming. There are many more indirect ways in which the lack of thinking harms one’s health as well.

Think, to understand what you want

Machines cannot reproduce desires and needs. Even in a hypothetical world where machines themselves have desire, machines cannot reproduce your specific desires and needs.

While knowing what you want sounds trivial, it usually requires thoughtful unpacking to understand what you truly want. For example, let’s say I feel irritated. Is it because I am in a hot subway station? Is it because I woke up late and couldn’t have breakfast and am now hungry? Is it because the person next to me is speaking loudly? Is it all of these or none of them? To disentangle this requires thought, and as long as LLMs do not have direct access to my brain, LLMs cannot provide the answers for me.

This principle, that LLMs cannot tell you what you need, generalizes beyond the self. Coding agents, for example, can generate code at superhuman speed, but they cannot operate without someone telling them what to do in the first place. Thus, an accurate understanding of what is needed, and having the thinking acumen to distill that, is a durable reason to think.

Think, to be original

A famous quip from Ted Chiang for LLMs is that they are a “blurry jpeg of the web”. That is, LLMs are a distillation of the monumental body of knowledge that humanity has compiled on the internet. While the individuals contributing to that knowledge were each unique, LLMs as a single model “average” over all of these individuals.

Being “average” might sound like a surefire path to mediocre results, but it is not. Consider the “average face”, which can be constructed by averaging the face images of a large number of people. Often, the “average” face is better-looking than each of the individual faces, a phenomenon known as averageness, which is used as way of quantitatively measuring beauty. Similarly, LLMs, which represent an “average” of cognition, have well-rounded capabilities few individuals can match.

The problem with “averageness” is thus more subtle. It is not that the “average” thinking LLMs can provide yields mediocre results: in fact it can deliver superior results than any individual on their own. Rather, the problem is that despite the superiority of the tool, you cannot stand out if you use LLMs for your thinking, because your superior tool will be something that everyone else has.

Meanwhile, your own experiences nudge you to think differently from everyone else, as they are inhabiting a different physical body. Thus, if you can think based on your experiences, which you cannot directly share with an LLM (yet), your thoughts will have a unique nature that others would not be able to replicate.

Think, for the joy of it

Yes, the cliché answer. I still think it’s true.

Addendum: Aaare my thoughts original, though?

I typed everything above with my own hands and with my own thoughts, without the influence of any LLM (sans proofreading). But I was curious how LLMs would answer my question of why I should continue thinking. Here is a what they had to say (I asked them to give a three point summary):

As an exercise, the reader can think about whether my thoughts are in fact original.

TLDR: This paper introduces LIBRO, a technique that automatically reproduces bug reports with high precision.

Motivation

When users or developers find bugs, they can report these bugs on issue tracker software, such as on GitHub as issues or on Jira. Such reports are called “bug reports”. Bug reports are generally written in natural language, and may be submitted without a test that reproduces the issue. This is a difficult situation for developers, as they will then have to manually try to reproduce the issue in the bug report by writing their own test code. Given the large volume of bug reports being submitted every day, this burden can be overwhelming. Thus, bug reproduction is a situation where automatic test generation could help developers.

Despite this, due to the difficulty of natural language processing, automatic bug reproduction has been mainly limited to crash bugs, where the expected behavior is clear - the program should not crash. This actually leaves some simple cases (from the human perspective) difficult to automatically handle for machines. For example, consider the following bug report from Commons Math:

NaN in “equals” methods

In “MathUtils”, some “equals” methods will return true if both argument are NaN. Unless I’m mistaken, this contradicts the IEEE standard.

If nobody objects, I’m going to make the changes.

In this case, it is clear what the expected behavior is to a human (equals methods should return false if both arguments are NaN). However, when trying to deal with this report using a rule-based algorithm, you immediately face two difficulties. First, the report does not explicitly spell out the expected behavior, so it is difficult to extract the expected behavior. Second, even if you get the oracle, all the code references in the report are disconnected, so it can be difficult to automatically write a test based on the available information. It is likely such difficulties that made automatic bug reproduction outside of crash bugs difficult.

Approach and Results

It is no longer news, but one of the observations of our paper is that large language models (LLMs) do quite a good job of overcoming these challenges. After all, LLMs are good enough at natural language processing to the extent that they are capable of explaining humor, and they are capable of generating code as well. Consequently, they are a natural solution to the test reproduction problem.

However, to build a tool that is actually useful to developers, simply generating many solutions from an LLM is not enough - we can’t hand a hundred candidate reproducing tests to a developer and say “job done”. Thankfully, for automatic bug reproduction, we find features that can help reduce this manual inspection cost. For example, a bug reproducing test should actually fail on the current, buggy version of the code, which allows us to automatically filter a portion of the unhelpful solutions. We bundled this heuristic, along with others, as an automated post-processing pipeline, specialized for handling bug reproduction.

With LLMs and the post-processing, which together we group in the tool LIBRO, we could both reproduce one-third of all of the bugs in the Defects4J benchmark, which is a substantially better performance than the baselines that we compared against. Furthermore, with the postprocessing pipeline we could achieve a precision of up to 0.85, reducing developer hassle with false positives.¹

Perspective A Year After Submitting The Paper

Since then, there has been a substantial amount of work on test generation using large language models. Testing does seem to be one of the most natural applications of LLMs for software engineering - while LLMs have the tendency to be wrong, it is okay for tests to be “wrong”, as this may actually lead to the testing of interesting behavior, as we note in Robert et al..

If I can leave one takeaway for any readers who have come this far, it is that while LLM results are certainly valuable, the generation results of LLMs can be much more valuable when they are evaluated by a rule-based system, as once this is done, it is easier for humans to have trust in such results. This is a theme of much of my subsequent work on LLMs: I argued this point in my GI workshop position paper, and explicitly made this a central theme of my Automatic Scientific Debugging.

Footnotes