Anyone in software engineering knows by now that being able to learn and get up to speed on new topics, libraries, or languages is a critical skill, both for onboarding onto a new team, as well as for continuously providing value on an existing team.
With the advent of fast, cheap, reasonably intelligent LLMs, efficient learning as a software engineer is more available than it’s ever been. However, not many people are taking advantage of it, either because they’re not aware of how to use the LLMs to their fullest potential, because their learning and meta-learning skills are under-trained, or perhaps both.
Here, we’ll walk through a few examples of how LLMs have allowed me to quickly fill skill gaps as they become relevant to my day-to-day work and side projects, to help demonstrate how you can also leverage LLMs to save time on keeping your skills sharp. These examples will also demonstrate how you can reframe the kinds of questions you ask while learning, in order to get higher-quality results.
I will be using Perplexity AI as my tool of choice for these demonstrations, as I think it offers the current best performance and UX for using LLMs to learn. This is because they’ve built a high-quality RAG system that does its best to enhance the LLM’s performance while weeding out probable low-quality sources from the eventual answer provided. For me, it has been a more or less drop-in replacement for Google or other search engines, but with significantly more utility and flexibility than a plain search engine.
Let’s start with an easy one. Many folks by now have tried simply asking about what a library, technology, or some other term is. I’ll use “What is Docker?” as an example since that’s something I suddenly needed to care about earlier in my career, when my output switched from being mostly-research to mostly-production.
“What is Docker?” is a great example of a bad question for an LLM, even a RAG-enhanced LLM like Perplexity AI’s system. The answer you get back is full of fluff and buzzwords and isn’t that helpful for understanding Docker in the contexts that are relevant to you. There is some good stuff here, but it’s buried amongst several tangential details that end up distracting you from the main points.
Let’s try again, but phrased somewhat differently. This time we’ll ask what problem Docker solves. Note that we get a clearer explanation now: Perplexity emphasizes that Docker solves the “works on my machine” problem and makes applications easier to package, test, and deploy in varied settings.
This is cool, and helps us in the here-and-now, but we want to learn over the long-term how to make sound technical decisions about which technologies to use and when. We want to actually understand Docker in context, and why it came to be so ubiquitous in 2020s software engineering.
This is where LLM-enhanced search really shines relative to either one on its own. We get the benefits of being able to ask follow-up questions with relevant conversational context, as well as the power of pulling in new sources of information as-needed.
Now, with this next answer, we get to map our mental model of all Docker’s concepts onto a tedious thing that people used to have to do manually.
react-hook-form is a good library for form state management in React. However, its documentation is pretty sparse on explanations, instead choosing to demonstrate its APIs by example. This is less than ideal if you’re not sure which APIs to use for which problems yet.
Compare react-hook-form’s own Get Started section to Perplexity AI’s explanation of its key abstractions:
Notice the following differences:
react-hook-form chooses to lead with an example that uses several of its APIs at once, forcing you to try to understand its specific way of doing field registration, form state validation and errors, and value-watching all at the same time. useForm is, and explains the respective roles of each method that it unpacks. It also provides a simpler example that is easier to follow.I don’t do this to pick on this library – like I said, it’s a good library that solves form problems as advertised. However, when I first started using it, I would have found a high-level summary of the abstractions more helpful for organizing my mental model relative to what was selected for the homepage and Get Started page of the documentation.
With LLMs, you can now reframe anything you like in the way that makes the most sense for how you learn.
Here’s another example. As a newbie to the library, I didn’t understand the difference between Controller and useController. In our codebase, we were already using both, and I needed to build on this existing code. The library’s documentation doesn’t directly address this question anywhere (although there are Stack Overflow and GitHub issue questions about it). So let’s ask Perplexity.
Now, I don’t love everything about this answer, because the distinction between “creating reusable controlled inputs” and “wrap[ping] non-primitive React components” wasn’t super clear from the examples provided (since they’re both just wrapping standard HTML <input>elements).
However, it does clearly (1) state that they are functionally equivalent, once at the beginning and once at the end, as well as (2) demonstrate using the same functionality from each, and how that needs to be declared depending on which one you’re working with.
This was enough for me to go back to our code and compare and contrast the usage there, and better understand the two APIs as a result.
Here, I have an example of an actual project that I worked on, which is a transcription app using OpenAI’s Whisper model running locally on CPU, using fastapi for the backend and React for the frontend.
First I ask Perplexity what I need to be familiar with while building such an application. Be prepared to ignore a lot of it, since some is just standard best-practices advice (“understand design,” “write tests,” etc.) that doesn’t need to be there in a super-alpha-MVP-v0 version of the app.
However, it did successfully identify a few things I didn’t sufficiently take into account when I did first start this project. For example,
I hadn’t really thought about those things at all. In my conception of the project, I expected the bulk of the work to be in other areas. But in reality, I spent at least a few hours fixing bugs and mistaken assumptions related to file transfer and CORS.
I’ll follow up by asking more about the file handling step, since that was one of the pieces that cost me the most time on this project, due to several different issues that came up. And in fact, Perplexity successfully highlights several issues that did come up due to gaps in my knowledge, particularly:
Now, it would be disingenuous to claim that simply by reading Perplexity’s answers, I would have been sufficiently prepared to handle these issues that came up. All of them required additional research on those areas to understand what my problem was and how to fix it.
However, it would have saved me a lot of time and frustration in the moment. Like you, I don’t have infinite time to work on side projects, and when I sit down to write code, I want to write code. Being more prepared upfront for the things I’d need to know, and potential problems I’d likely run into, would have saved me a lot of time being stuck on one bug or another.
The biggest thing is that I would have had the chance to learn about them in free moments on other occasions, instead of having to context-switch in the middle of the feature work (and thus interrupt my flow) to go and get up to speed on something.
Lastly, asking about need-to-know info up front gives you a better idea of how much time the project is likely to take, since you will have identified at the beginning whether you have knowledge gaps and how significant they are.
As we wrap up, here are a few more rapid-fire tips to help you get the most out of these tools and services.
Tip: Use the best model you have access to
Stats from OpenAI’s blog post releasing GPT-4 indicate that GPT-4 is often between 10-20% more accurate, depending on field, than GPT-3.5 in providing answers.
If you’re still only using GPT-3.5 and you’re a software engineer, I can imagine you’re not particularly impressed with what you’ve seen. Try again with access to GPT-4, Claude 2, or another model that’s closer to topping the present-day leaderboards, and see where you get.
Keep up on the latest releases, and note that in the last couple years, big jumps in performance have arrived as often as every 3-6 months.
Tip: Focus on fundamentals, relationships, and high-level information
Use LLMs for answers about fundamentals, relationships and comparisons, and “high-level” information that helps you better structure your knowledge, not specific facts.
If you do get specific facts from an LLM, you will still have to double-check them against Wikipedia, source code, or another ground-truth source. While hallucination is less of a problem nowadays, with GPT-4 hallucinating significantly less often than 3.5 does (see above), it’s still rather frequent for LLMs to invent facts, APIs, and information that seems like it should be true, even if it doesn’t actually exist.
Particularly when searching for info about a library works, if you can’t find a straightforward answer about implementation details in the docs, you’re better off reading the source code than asking an LLM, unless you’re using a specialized tool with direct access to the codebase you’re asking about (e.g. Sourcegraph Cody).
Tip: Use Custom Instructions
Imagine you’re an all-knowing oracle, fielding questions from many little curious entities, but you have zero visibility into the background or experience of any of those entities tossing questions at you. It’d be rather difficult to provide an appropriate answer if you can’t make certain assumptions about what they know already, right?
If the system you’re using allows for custom instructions, or any sort of profile about yourself, actually filling that section out can be incredibly helpful for getting higher-quality responses. Here’s a repurposed template from my own AI Profile on Perplexity that you can copy-paste and fill out if you haven’t already. OpenAI also provides this option for ChatGPT users.
“I’m a [mid]-level [full-stack] software engineer. I live in [the Bay Area].
[Info on your background, what you know and don’t know.] I transitioned into engineering from a non-computer science background. I previously studied psychology at Stanford, and I later decided to get into data science and machine learning by self-teaching. I worked for a couple years on NLP in the early transformer days (on BERT models and such), then transitioned to SWE when ML got boring. Now I work at a company that builds [whatever your company does].
[Preferences on how the LLM should respond to you.] For general questions, please answer very thoroughly and to the best of your ability. I almost always prefer more detail to less detail. For very targeted and specific questions, provide a concise answer, but leave openings for elaboration and discussion. Freely use advanced and technical language as long as any new concepts are explained. [Any additional context on what operating system you use, (programming or speaking) languages you know and use, etc.]”
There are plenty of other creative ways to use LLMs to learn, but these are some of the most straightforward and common ways that I’ve used them, which have provided outsized returns for the level of effort. Learning to frame questions to LLMs effectively can be like having a personal tutor for any subject you care about, but at a fraction of the usual cost.
For more on LLMs, developing LLM-based products, LLM learning products I recommend, and differences between today’s and yesterday’s LLMs, check out this interview our author Jessica did with me late last year.
