Using LLMs to Learn 3x Faster As An Engineer

A woman software engineer pair programming with a robot (metaphor for an LLM)

Anyone in software engineering knows by now that being able to learn and get up to speed on new topics, libraries, or languages is a critical skill, both for onboarding onto a new team, as well as for continuously providing value on an existing team. 

With the advent of fast, cheap, reasonably intelligent LLMs, efficient learning as a software engineer is more available than it’s ever been. However, not many people are taking advantage of it, either because they’re not aware of how to use the LLMs to their fullest potential, because their learning and meta-learning skills are under-trained, or perhaps both. 

Here, we’ll walk through a few examples of how LLMs have allowed me to quickly fill skill gaps as they become relevant to my day-to-day work and side projects, to help demonstrate how you can also leverage LLMs to save time on keeping your skills sharp. These examples will also demonstrate how you can reframe the kinds of questions you ask while learning, in order to get higher-quality results.

I will be using Perplexity AI as my tool of choice for these demonstrations, as I think it offers the current best performance and UX for using LLMs to learn. This is because they’ve built a high-quality RAG system that does its best to enhance the LLM’s performance while weeding out probable low-quality sources from the eventual answer provided. For me, it has been a more or less drop-in replacement for Google or other search engines, but with significantly more utility and flexibility than a plain search engine.

Example 1: Quick overview of unfamiliar areas

Let’s start with an easy one. Many folks by now have tried simply asking about what a library, technology, or some other term is. I’ll use “What is Docker?” as an example since that’s something I suddenly needed to care about earlier in my career, when my output switched from being mostly-research to mostly-production.

“What is Docker?” is a great example of a bad question for an LLM, even a RAG-enhanced LLM like Perplexity AI’s system. The answer you get back is full of fluff and buzzwords and isn’t that helpful for understanding Docker in the contexts that are relevant to you. There is some good stuff here, but it’s buried amongst several tangential details that end up distracting you from the main points.

Let’s try again, but phrased somewhat differently. This time we’ll ask what problem Docker solves. Note that we get a clearer explanation now: Perplexity emphasizes that Docker solves the “works on my machine” problem and makes applications easier to package, test, and deploy in varied settings.

This is cool, and helps us in the here-and-now, but we want to learn over the long-term how to make sound technical decisions about which technologies to use and when. We want to actually understand Docker in context, and why it came to be so ubiquitous in 2020s software engineering.

This is where LLM-enhanced search really shines relative to either one on its own. We get the benefits of being able to ask follow-up questions with relevant conversational context, as well as the power of pulling in new sources of information as-needed.

Now, with this next answer, we get to map our mental model of all Docker’s concepts onto a tedious thing that people used to have to do manually. 

  • Docker images’ role is to provide environment consistency, decreasing the amount of time people spend debugging small differences between servers that cause install and runtime errors. Setup is now automated, consistent, and more reliable.
  • Docker containers are a more efficient iteration on VMs, with each container able to logically act as a separate host, but without having to keep a copy of an entire operating system on each one.
  • Stepping back, we can also notice that the standardized set of primitives Docker has chosen to provide this functionality is portable. It’s portable both in the sense of being able to generalize it to many different deployment environments (different machine types, cloud hosting providers, and so on), as well as in the sense of it being a generalizable “learn-once, use-on-every-team” skill that developers can take with them, rather than having to memorize every bespoke process the current employer or team is using.

Example 2: Understand the key abstractions of a library so you can use it effectively

react-hook-form is a good library for form state management in React. However, its documentation is pretty sparse on explanations, instead choosing to demonstrate its APIs by example. This is less than ideal if you’re not sure which APIs to use for which problems yet.

Compare react-hook-form’s own Get Started section to Perplexity AI’s explanation of its key abstractions:

Notice the following differences:

  • react-hook-form chooses to lead with an example that uses several of its APIs at once, forcing you to try to understand its specific way of doing field registration, form state validation and errors, and value-watching all at the same time. 
  • Meanwhile, Perplexity begins by actually telling you what useForm is, and explains the respective roles of each method that it unpacks. It also provides a simpler example that is easier to follow.

I don’t do this to pick on this library – like I said, it’s a good library that solves form problems as advertised. However, when I first started using it, I would have found a high-level summary of the abstractions more helpful for organizing my mental model relative to what was selected for the homepage and Get Started page of the documentation.

With LLMs, you can now reframe anything you like in the way that makes the most sense for how you learn.

Here’s another example. As a newbie to the library, I didn’t understand the difference between Controller and useController. In our codebase, we were already using both, and I needed to build on this existing code. The library’s documentation doesn’t directly address this question anywhere (although there are Stack Overflow and GitHub issue questions about it). So let’s ask Perplexity.

Now, I don’t love everything about this answer, because the distinction between “creating reusable controlled inputs” and “wrap[ping] non-primitive React components” wasn’t super clear from the examples provided (since they’re both just wrapping standard HTML <input> elements). 

However, it does clearly (1) state that they are functionally equivalent, once at the beginning and once at the end, as well as (2) demonstrate using the same functionality from each, and how that needs to be declared depending on which one you’re working with. 

This was enough for me to go back to our code and compare and contrast the usage there, and better understand the two APIs as a result.

Example 3: Prepare for what to expect when you begin a new project

Here, I have an example of an actual project that I worked on, which is a transcription app using OpenAI’s Whisper model running locally on CPU, using fastapi for the backend and React for the frontend. 

First I ask Perplexity what I need to be familiar with while building such an application. Be prepared to ignore a lot of it, since some is just standard best-practices advice (“understand design,” “write tests,” etc.) that doesn’t need to be there in a super-alpha-MVP-v0 version of the app. 

However, it did successfully identify a few things I didn’t sufficiently take into account when I did first start this project. For example,

  • Asynchronous requests (because a synchronous request will time out for a sufficiently big file, async is also better UX)
  • File handling (correctly sending a large binary file from frontend to backend, extracting audio from video using ffmpeg, and dealing with filesystem permissions)
  • Figuring out CORS 

I hadn’t really thought about those things at all. In my conception of the project, I expected the bulk of the work to be in other areas. But in reality, I spent at least a few hours fixing bugs and mistaken assumptions related to file transfer and CORS.

I’ll follow up by asking more about the file handling step, since that was one of the pieces that cost me the most time on this project, due to several different issues that came up. And in fact, Perplexity successfully highlights several issues that did come up due to gaps in my knowledge, particularly:

  • Correctly handling multipart/form-data for very big media files (anywhere from a few MB to a few hundred MB) – Cost me 1-2 hours
  • Preprocessing: extracting the audio stream from a video file using ffmpegCost me a few hours to work out how I would handle ensuring the user has ffmpeg available, as well as locating the correct ffmpeg installation, as well as learning about types of media storage formats to correctly invoke the ffmpeg commands
  • Doing pathing and permissions correctly per operating system, since I was building the app on MacOS but it needed to work the same way on Windows – Cost another couple hours to learn about and understand after bumping up against it, but thankfully was easy to solve thanks to a readily available Python library

Now, it would be disingenuous to claim that simply by reading Perplexity’s answers, I would have been sufficiently prepared to handle these issues that came up. All of them required additional research on those areas to understand what my problem was and how to fix it. 

However, it would have saved me a lot of time and frustration in the moment. Like you, I don’t have infinite time to work on side projects, and when I sit down to write code, I want to write code. Being more prepared upfront for the things I’d need to know, and potential problems I’d likely run into, would have saved me a lot of time being stuck on one bug or another. 

The biggest thing is that I would have had the chance to learn about them in free moments on other occasions, instead of having to context-switch in the middle of the feature work (and thus interrupt my flow) to go and get up to speed on something.

Lastly, asking about need-to-know info up front gives you a better idea of how much time the project is likely to take, since you will have identified at the beginning whether you have knowledge gaps and how significant they are.

Final Tips

As we wrap up, here are a few more rapid-fire tips to help you get the most out of these tools and services.

Tip: Use the best model you have access to

Stats from OpenAI’s blog post releasing GPT-4 indicate that GPT-4 is often between 10-20% more accurate, depending on field, than GPT-3.5 in providing answers. 

A grouped bar chart with category on the x-axis, accuracy on the y-axis, demonstrating that gpt-4 outperforms the next-best chatgpt model by 10-25% depending on the category.

If you’re still only using GPT-3.5 and you’re a software engineer, I can imagine you’re not particularly impressed with what you’ve seen. Try again with access to GPT-4, Claude 2, or another model that’s closer to topping the present-day leaderboards, and see where you get. 

Keep up on the latest releases, and note that in the last couple years, big jumps in performance have arrived as often as every 3-6 months.

Tip: Focus on fundamentals, relationships, and high-level information

Use LLMs for answers about fundamentals, relationships and comparisons, and “high-level” information that helps you better structure your knowledge, not specific facts. 

If you do get specific facts from an LLM, you will still have to double-check them against Wikipedia, source code, or another ground-truth source. While hallucination is less of a problem nowadays, with GPT-4 hallucinating significantly less often than 3.5 does (see above), it’s still rather frequent for LLMs to invent facts, APIs, and information that seems like it should be true, even if it doesn’t actually exist. 

Particularly when searching for info about a library works, if you can’t find a straightforward answer about implementation details in the docs, you’re better off reading the source code than asking an LLM, unless you’re using a specialized tool with direct access to the codebase you’re asking about (e.g. Sourcegraph Cody). 

Tip: Use Custom Instructions

Imagine you’re an all-knowing oracle, fielding questions from many little curious entities, but you have zero visibility into the background or experience of any of those entities tossing questions at you. It’d be rather difficult to provide an appropriate answer if you can’t make certain assumptions about what they know already, right?

If the system you’re using allows for custom instructions, or any sort of profile about yourself, actually filling that section out can be incredibly helpful for getting higher-quality responses. Here’s a repurposed template from my own AI Profile on Perplexity that you can copy-paste and fill out if you haven’t already. OpenAI also provides this option for ChatGPT users.

“I’m a [mid]-level [full-stack] software engineer. I live in [the Bay Area].

[Info on your background, what you know and don’t know.] I transitioned into engineering from a non-computer science background. I previously studied psychology at Stanford, and I later decided to get into data science and machine learning by self-teaching. I worked for a couple years on NLP in the early transformer days (on BERT models and such), then transitioned to SWE when ML got boring. Now I work at a company that builds [whatever your company does].

[Preferences on how the LLM should respond to you.] For general questions, please answer very thoroughly and to the best of your ability. I almost always prefer more detail to less detail. For very targeted and specific questions, provide a concise answer, but leave openings for elaboration and discussion. Freely use advanced and technical language as long as any new concepts are explained. [Any additional context on what operating system you use, (programming or speaking) languages you know and use, etc.]”

Conclusion

There are plenty of other creative ways to use LLMs to learn, but these are some of the most straightforward and common ways that I’ve used them, which have provided outsized returns for the level of effort. Learning to frame questions to LLMs effectively can be like having a personal tutor for any subject you care about, but at a fraction of the usual cost.

For more on LLMs, developing LLM-based products, LLM learning products I recommend, and differences between today’s and yesterday’s LLMs, check out this interview our author Jessica did with me late last year.

https://charlenechambliss.com

Charlene is a senior software engineer at Aquarium, where she leverages her varied background in data analysis, traditional and modern NLP, and web development to work on Tidepool, helping makers of LLM-based products understand how to best improve their product's UX, stickiness, and monetization. She also likes to create content and speak publicly now and then, from writing on her personal website to co-hosting a webinar with the Director of Science while at Primer, as well as appearing on several panels and podcasts (including TWIML AI).