When initially experimenting with LLMs and agentic AI, software program engineers at Notion AI utilized superior code era, complicated schemas, and heavy instructioning.
Shortly, although, trial and error taught the staff that it might do away with all of that difficult information modeling. Notion’s AI engineering lead Ryan Nystrom and his staff pivoted to easy prompts, human-readable representations, minimal abstraction, and acquainted markdown codecs. The end result was dramatically improved mannequin efficiency.
Making use of this re-wired method, the AI-native firm launched V3 of its productiveness software program in September. Its notable characteristic: Cutomizable AI brokers — which have shortly grow to be Notion’s most profitable AI device thus far. Based mostly on utilization patterns in comparison with earlier variations, Nystrom calls it a “step function improvement.”
“It's that feeling of when the product is being pulled out of you rather than you trying to push,” Nystrom explains in a VB Past the Pilot podcast. “We knew from that moment, really early on, that we had something. Now it's, ‘How could I ever use Notion without this feature?’”
‘Rewiring’ for the AI agent period
As a standard software program engineer, Nystrom was used to “extremely deterministic” experiences. However a lightweight bulb second got here when a colleague suggested him to easily describe his AI immediate as he would to a human, quite than codify guidelines of how brokers ought to behave in numerous situations. The rationale: LLMs are designed to grasp, “see” and purpose about content material the identical approach people can.
“Now, whenever I'm working with AI, I will reread the prompts and tool descriptions and [ask myself] is this something I could give to a person with no context and they could understand what's going on?” Nystrom stated on the podcast. “If not, it's going to do a bad job.”
Stepping again from “pretty complicated rendering” of information inside Notion (comparable to JSON or XML) Nystrom and his staff represented Notion pages as markdown, the favored device-agnostic markup language that defines construction and which means utilizing plain textual content with out the necessity for HTML tags or formal editors. This permits the mannequin to work together with, learn, search and make adjustments to textual content information.
Finally, this required Notion to rewire its methods, with Nystrom’s staff focusing largely on the middleware transition layer.
In addition they recognized early on the significance of exercising restraint in terms of context. It’s tempting to load as a lot info right into a mannequin as attainable, however that may sluggish issues down and confuse the mannequin. For Notion, Nystrom described a 100,000 to 150,000 token restrict because the “sweet spot.”
“There are cases where you can load tons and tons of content into your context window and the model will struggle,” he stated. “The more you put into the context window, you do see a degradation in performance, latency, and also accuracy.”
A spartan method can be essential within the case of tooling; this might help groups keep away from the “slippery slope” of countless options, Nystrom suggested. Notion focuses on a “curated menu” of instruments quite than a voluminous Cheesecake Manufacturing facility-like menu that creates a paradox of alternative for customers.
“When people ask for new features, we could just add a tool to the model or the agent,” he stated. However, “the more tools we add, the more decisions the model has to make.”
The underside line: Channel the mannequin. Use APIs the best way they had been meant for use. Don't attempt to be fancy, don't attempt to overcomplicate it. Use plain English.
Take heed to the complete podcast to listen to about:
Why AI continues to be within the pre-Blackberry, pre-iPhone period;
The significance of "dogfooding" in product improvement;
Why you shouldn’t fear about how price efficient your AI characteristic is within the early levels — that may be optimized later;
How engineering groups can hold instruments minimal within the age of MCP;
Notion’s evolution from wikis to full-blown AI assistants.
Subscribe to Past the Pilot on Apple Podcasts, Spotify, and YouTube.
