Please, Sir, I Want Some Lore.

My favourite genre of YouTube as a kid was “making of” videos. These were often produced by educational, science TV channels and would guide the viewer through how some household object was made: pencils, toasters, windows. They’d walk through every step of production, slowly panning across gleaming factories and zooming in close-up to repetitive machining. I couldn’t get enough of these clips. I’m probably yet to fully appreciate the impact they left on my psyche.

Something that was missing from these videos, though, was more personal history. I like understanding not just how something works, but why. I was missing intuition for certain design choices; I wished they could have presented science-as-a-story, alongside plain science-as-fact. I wanted a more colourful history: I would have loved to see the 5x-great-granddaughter of the company founder come out to read diary entries about why he started milling pencils in 1822 because one day he ran out of ink and couldn’t write a letter to his childhood sweetheart.

I also grew up watching Behind-The-Scenes cuts of movies that I’d never watched, particularly stop-motion features. I liked these videos more. They’d interleave cinematographic shots of animators hard at work with commentary from the director or producers. I got a better sense of the emotions they were trying to convey and of specific filming challenges. I revelled in the stories when they’d get into some despairing disaster, only for some intern to stumble upon the perfect solution. Or, other times, an interviewer would ask why something was shot a certain way, and they’d just laugh and say it was totally random. There were a lot of compelling anecdotes, so it’s really not surprising I felt more drawn to these videos. What do you get when you cross encyclopaedic curiosity with a humdrum, otherwise gossip-free life? A need for lore.

By lore, I mean the backstory of why something is done, the little stories that shaped decisions, and the reason why a concept is presented a certain way. Lore is when your colleague offhandedly mentions the personal altercation that contributed to their current vendetta against a particular research group. Lore is when your supervisor tells tales about a formative internship. Lore is when your friend opens up about why their current thesis pivot means so much more to them.

Lore is important for doing real science, and even more crucially, learning how to do it. The traditional way to do this is via academia, but the ivory tower seems poised to crack, what with funding cuts and increasingly prevalent independent research. As scientific inquiry transitions into new alternative forms, research communication is getting more room to grow. I think we should fill this space with lore.

Training Season’s Over

I recently read Dwarkesh Patel’s The Scaling Era on the last half-decade of AI research. It frames itself as an “oral history”: this already gets it some brownie points as a lore drop. It collects interview snippets from lab leads and researchers at the usual suspects, including OpenAI, Anthropic, DeepMind. In a chapter about scaling, Gwern Branwen shares a particularly relevant note on the research process:

Another issue is that they made the basic error I had made, thinking that algorithms are more important than compute. That’s partly due to a systematic falsification of the actual origins of ideas in the research literature. Papers do not tell you where ideas come from; they just tell you a nice-sounding story about how something was discovered. So even if you appreciate the role of trial and error and compute in your own experiments, you probably think, “I got lucky. Over in the next lab, they do things with the power of thought and deep insight.” But it turns out that everywhere you go, compute, trial and error, and serendipity play enormous roles in how things actually happen.

This bit about papers glossing over nice-sounding stories really spoke to me. I’m disappointed that there’s often not a channel for authors to expand on their process in more detail, particularly the wrong turns and challenges before they zeroed in on their final approach. I took a grad course this last term, and my favourite discussions were those where we covered the professor’s own papers. She added extra context on why certain approaches were chosen as baselines and why certain plots were extra funky and sometimes showed us the differences between the original submission and final camera-ready. This deepened my understanding not just of the paper, but of the academic zeitgeist it was born in at the time. It’s a bit like going to a museum: interpreting the piece on your own gives you some impression, but you always learn something more from the description placard, especially if the artist or research team wrote it themselves. However, we rarely get these details in a conventional paper format.

As Branwen alludes to, this sanitization of the raw stories behind research also affects the individual scientist’s learning. For one, it can play to imposter syndrome and is a bit reductionist. On the other hand, maybe there’s something in someone else’s luck and trial and error that you might have been able to explain or build on. It would be nice if we reified that serendipity to share it more broadly, so that others might also be able to benefit.

How else are we meant to train the next generation of researchers? Everyone tells me that a PhD is mostly not about the work, but about the journey of understanding how to do research. This requires a salubrious environment in which to learn the ropes: when to try this approach, when to visualize data like so. Today, this gets verbally passed down, like ancestral knowledge. One only has so many supervisors and colleagues, though, so there’s plenty of interesting background behind impactful work that inevitably gets lost.

Eliezer Yudkowsky puts it better in The Scaling Era:

We don’t have a systematic training method for producing real science in that sense. A quarter of Nobel laureates are the students of other Nobel laureates because we never figured out how to teach science. We have an apprentice system instead. We have people who pick out people who they think can be scientists and hang around them in person, and something we’ve never written down in a textbook passes down. That’s where the revolutionaries come from. There are whole countries trying to invest in having scientists. They churn out people who write papers. None of it goes anywhere because the part that was legible to the bureaucracy is, have you written the paper? Can you pass the test? This is not science.

As I experience more of the research field, I’m beginning to notice the influences of this paper-factory culture that Yudkowsky rightfully decrees as “not science”. In my lab, we’ve had discussions about how AI helping to accelerate research makes it even more important to choose the right problems. As my labmate said, we shouldn’t just “go nowhere, but faster”. Now that the bottleneck has shifted to ideation instead of execution, assessing a potential research direction is a matter of taste. Taste is most easily refined via imitation: I think the easiest way to get a sense of someone’s taste is to work with them, and the next best thing is to read into their decisions and ask why. This is where understanding the backstory of certain choices can be extremely helpful, and hence, why recording more context is necessary.

Obviously, you can’t do groundbreaking research if you just copy the types of decisions someone else made: that’s how you get stuck in old-school research. But learning to do research is like finetuning one’s own taste neural network, and you can’t train a model without data being available.

Getting Down to Business

Effective research training should be the main reason to valorise lore. In terms of other incentives, though, lore can fit into existing structures. In environments where research output needs to be (overly) quantified, slipping more lore into a publication or presentation still means they check the same boxes and increment the same counters. (I’d argue it makes communicating the same work more fun: this way you get more juice out of the same metaphorical beans.) In a broader sense, lore plays on similar social motivations as the open-access movement. For one, it promotes a warm-and-fuzzy sense of altruistic transparency. For another, it attracts others to the work. Lore can be a more equitable and scalable form of support, as it helps spread implicit research knowledge more broadly instead of keeping it concentrated in exclusive, elite labs.

A more late-stage-capitalistic motivation might be that lore is an underexploited source of training data for research-focused agents. For example, OpenAI’s GPT-Rosalind is intended to aid drug discovery: I’d imagine it’d benefit from reading other researcher’s drug development process notes. There’re startups like Mercor, Datacurve, and AfterQuery already working in this data curation space. It’s probable the Bitter Lesson might mean this data isn’t actually all that useful, but on the other hand learning from others’ lore-like reasoning trajectories is what conventional PhDs boil to down to after all.

We should also be careful not to attach too many incentives. The community would be even worse off if research histories were disingenuously manufactured, as we’d be educating people on shoddy priors. We should be okay with saying, sometimes, that there was no lore. Maybe a decision really was made on a whim. This still gives us some value, though: maybe we learn that in this field, random choices can lead to powerful results¹. We should also be okay disclosing we don’t know why something worked, or that our first ten really good ideas failed. We should appreciate the intent behind the disclosure , and not overly so its actual contents.

I’m confident we can get the motivation right. Arcadia, a “science-first company” working in biotech, is an early example of lore working out; so is Astera. In a lovely piece for Asterisk Mag, Jolie Gan writes:

In 2025, Arcadia’s CEO, Seemay Chou, announced that the company would no longer fund traditional journal publications. Instead, Arcadia’s scientists publish their research as modular outputs, including partial results, methods, negative findings, and works-in-progress, complete with datasets, methods, and commentary on what worked and what didn’t.

This is exactly what I refer to as lore, and it’s refreshing to see this as a conscious decision for a research institution. Interestingly, Arcadia is for-profit and was funded via (billionaire) angel investments. This highlights that lore and transparency in research don’t have to come at odds with financial viability.

Beyond acmart

As Yudkowsky mentioned, paper-writing is one of the most legible markers of productivity for bureaucratic ends, and I don’t think that’ll change any time soon. Yet as I said above, even in conventional research journals and conferences, we can sprinkle in lore about our work without eschewing the standard 12-page format. One of my favourite papers I’ve read recently is “Simplifying and Isolating Failure-Inducing Input” (the Delta Debugging paper). The technique is neat, the formalism is extremely clean, and, most importantly, they explain the intuition behind their decisions and evaluation. Even more exciting: they published a retrospective, chock-full of lore. I particularly enjoyed the tongue-in-cheek references to industry partners who-may-not-be-named and the historical tie-ins that set the scene for how the approach evolved. Both these works were published via traditional channels and would fit the criteria for any publication bean-counters, yet both do a great job of sharing lore. I would very much like to see more retrospectives like this for other papers².

There’s a current shift away from traditional academic institutions too, and this makes new space for lore. With the rise of independent research and publishing venues like Distill, Twitter, and personal sites, we can embed chatty podcasts covering rationale and collect Gossip-Girl-esque photo blasts of previous figure iterations. These wouldn’t fit into conventional papers but add so much colour to the research process.

We can also consider entirely different ways to communicate research. I’m a fan of the work CultRepo does to produce documentaries on the histories of open-source tech projects and key community members. They feel like my beloved childhood “behind the scenes” videos, but for tooling and frameworks I hear about and use every day. Their work is insanely high-quality and in-depth, with the added bonuses of being very consumable and viral. Their films are an excellent example of how we can make lore accessible and interesting, and their popularity underscores that there’s definitely appetite for it.

Conclusion

We might be in a “researcher fertility crisis”, and as Branwen and Yudkowsky touch on, I think the current culture around papers will exacerbate it. As a counterbalance, we’d better figure out how to transmit research taste before the more senior scientific cohort retires out. I’m certain more in-depth lore is part of the answer. Organizations like Arcadia are social proof that supplying lore is viable, and the demand’s there for new avenues of communicating research stories as well. I promise I’m not just fishing for research gossip: lore is essential for training taste and supporting scientific progress as a whole.

In this post I’ve repeatedly referred to the shift away from traditional research institutions, but even within academia, there are growing initiatives to improve transparency. For instance, for CS conferences it’s now quite common to have artifact evaluations and experiment reproduction setups. It’s not quite to the scale of what I’d want to see, since it doesn’t require articulating why something works, but at least folks can now see into the how. There are also often “experience” tracks for folks to describe the actual implementation and implications of a technique put into practice. I get the impression this usually focuses more on contemporary papers instead of historical seminal work, but again is also a good first step.

I blog about my research so I can write the types of backstory-filled articles I’d like to read. I want to see more lore out there, though, so:

• If you’re in the research field and writing a paper, include more context, especially in your introduction and evaluation sections. Even better, write a companion blog post for all the anecdotes that wouldn’t be appropriate for a typical paper. Tell the story of how this work came to be. Look through your research logbook for key tidbits that shaped its direction. Hell, go through your Close Friends story on Instagram to see what rants you posted.
• If you lead a research team, you are much too important to be reading my blog, but encourage your colleagues to discuss and share intermediate results. Nudge them to keep research logs, ideally in public channels. Laud failures or challenges as well as successes, and foster a culture where less-exciting results get communicated somewhere too.
• If you’re a journal editor, you are also too important to be here, but if you’re looking for suggestions, I’d love to see more retrospective pieces, even for recent works or works-in-progress. While space-constrained conference formats might not be ideal for this type of historical context, it seems journals offer a bit more breathing room for sharing these stories.

I hope the New Age of research communication enables much more lore to be shared and much better science to be done. Now if you’ll excuse me, I have more “How It’s Made” queued up to watch.