Key Points From This Episode:

• Drew and his co-founders’ background working together at RJ Metrics.
• The lack of existing data solutions for Amazon Redshift and how they started dbt Labs.
• Initial adoption of dbt Labs and why it was so well-received from the very beginning.
• The concept of a semantic layer and how dbt Labs uses it in conjunction with LLMs.
• Drew’s insights on a recent paper by Apple on the limitations of LLMs’ reasoning.
• Unpacking examples where LLMs struggle with specific questions, like math problems.
• The importance of thoughtful prompt engineering and application design with LLMs.
• What is needed to maximize the utility of LLMs in enterprise settings.
• How understanding the specific use case can help you get better results from LLMs.
• What developers can do to constrain the search space and provide better output.
• Why Drew believes prompt engineering will become less important for the average user.
• The exciting potential of vector embeddings and the ongoing evolution of LLMs.

Quotes:

“Our observation was [that] there needs to be some sort of way to prepare and curate data sets inside of a cloud data warehouse. And there was nothing out there that could do that on [Amazon] Redshift, so we set out to build it.” — Drew Banin [0:02:18]

“One of the things we're thinking a ton about today is how AI and the semantic layer intersect.” — Drew Banin [0:08:49]

“I don't fundamentally think that LLMs are reasoning in the way that human beings reason.” — Drew Banin [0:15:36]

“My belief is that prompt engineering will – become less important – over time for most use cases. I just think that there are enough people that are not well versed in this skill that the people building LLMs will work really hard to solve that problem.” — Drew Banin [0:23:06]

Links Mentioned in Today’s Episode:

Understanding the Limitations of Mathematical Reasoning in Large Language Models

Drew Banin on LinkedIn

dbt Labs

How AI Happens

Sama