Learning, Understanding, and Managing Knowledge for LLM-Backed Application Development
I am learning about GenAI while building out a schema for all the components involved in these types of application at the same time. A true case of "building the plane as it rolls down the runway"
In my last post, I explored how I'm trying to speed up my learning in Generative AI (GenAI) development by using high-abstraction tools, such as visual workflow builders and low-code/no-code platforms. These tools simplify the process by allowing me to focus on building functionality without getting bogged down in lower-level coding tasks, enabling faster prototyping and experimentation. They also help me concentrate on the overall approach to solving a business problem, rather than getting lost in the syntax and semantics of different tools. To be clear, I'll still dive into lower-level coding when needed, but for now, I'm focused on skipping boilerplate and configuration coding.
I've started experimenting with n8n, a low-code workflow automation tool, to create workflows that automate email newsletter content parsing, summarization, and named entity extraction (my GenAI "Hello World" project, as it were). Through this process, I've found that n8n does indeed work well as a playground for my LLM experiments. Instead of focusing solely on specific implementations of the various technologies in a tech stack, I'm using n8n to directly engage with different LLMs and determine which can consistently extract and summarize content from email newsletters. This shift allows me to streamline my approach while learning the capabilities of a wide array of LLMs in practical hands-on scenarios.
Information overload
Information overload has been my biggest challenge, so I'm making it a top priority to address. The pace of development in the LLM ecosystem is relentless—new models, new tools, improved capabilities, and changing costs all make it nearly impossible to manually keep track of the available LLMs. As an engineer learning how to build LLM-backed applications, it's clear that before I even attempt to understand LLM capabilities, I need a system to manage the information about them.
That's why I'm pausing my deep dive into specific LLMs to build a knowledge management system that brings information from newsletters, blogs, and community updates directly to me in a digestible form. My aim is to automatically collect, summarize, and organize the latest updates on GenAI and LLMs. Using tools like n8n, I plan to automate this information flow—ingesting incoming emails, summarizing their contents, extracting named entities, and creating a consolidated daily update for myself. This will serve as my foundation, a steady intake of LLM-related knowledge that will ultimately enable me to make informed decisions on which models to use and when.
Abstraction is key
Another realization this week is the need for flexibility. Whatever LLM stack I assemble must be able to adapt as new models and better technologies emerge. This means having a way to swap LLMs and other components in and out seamlessly, and using agentic workflows to leverage multiple LLMs, each specialized in its own domain. In practice, this could look like one LLM coordinating tasks while other, specialized LLMs handle more niche functions.
The journey is not just about understanding what LLMs exist, but also about creating a robust, semi-automated system for tracking and understanding this rapidly evolving landscape. With this foundation, I hoping to be able to explore LLMs a bit more confidently, knowing I have the information I need to stay on top of developments—without drowning in the noise.
Developing a schema for LLM-backed applications
Another focus this week has been developing a schema to outline the essential components needed for building LLM-backed applications. To start fleshing out this schema, I used a couple of resources. The first is an article from Andreessen Horowitz on emerging LLM stacks from last year, which provides a detailed overview of the components and data flow involved in LLM-backed systems. I'm also looking at LangChain’s orchestration tools, specifically their Components, which offer a structured breakdown of the key elements necessary for coordinating tasks within LLM applications.
By combining these two sources, I’ve begun developing a more simplified and practical schema to help me (and possibly others) understand and keep tabs on the evolving options for the core components of LLM-backed applications. This schema is still in development, but I'll share more details in a future post.
The meta-meta
Ongoing reflections on GenAI-assisted learning and writing, "learning in public," and the topic of GenAI application development.
This journey of learning and understanding LLM-backed applications is also serving a dual purpose as an exploration of how generative AI can assist in my writing process.
Leveraging tools like n8n and various LLMs, I'm working on new ways to enhance my learning workflows, experiment more efficiently, and document my learning in real time. It feels like building a course you want to teach, using content about a technology that was, in some cases, released just a week before. Craziness.
I've also started using OpenAI's new Canvas tool to help with the writing process, which has provided an additional layer of support and flexibility in drafting and refining my ideas.
The rapid change in this field of software development—both in the tools used to build applications in general and in the stacks for building LLM-backed applications—is moving so fast that by the time I finish an article, my thinking has already evolved. It's a bit of insanity... but the kind I fortunately love.
These reflections will continue to evolve as I build out this larger body of work in this Substack, sharing both the challenges and the innovations that emerge along the way. I already feel like I need to bump up to yet another higher level of abstraction to keep on top of this field.