Pulling Things Together

We’ve gone through most of the steps manually that can be used to solve the original challenge. This post will discuss how we can bring them all together and what we’ve learned through this process.

Routing Logic

Now that values are stored in a list, we can pass the list to our natural language SQL query. We can alter our function so that it permits passing a python list along with the original question to the BigQuery function.

Function Definition

Ensure the bq_qna function supports passing the question as well as a list to test against.

# @title Run a sample join query with mascots that can fly (via a list)

#Testing 1

bq_qna('Show me are all the mbb team markets that have a mascot in this list? List: {matched_test}'.format(matched_test=true_list))

SQLQuery:SELECT DISTINCT mbb_teams.market FROM mbb_teams JOIN mascots ON mbb_teams.market = mascots.market WHERE mascots.mascot IN ('Peacock', 'Rooster', 'Blue Jay', 'Winged Blue Horse')

SQLResult: [('Middle Tennessee',), ('South Carolina',), ('Jacksonville State',), ('Coastal Carolina',), ("St. Peter's",), ('Creighton',)]

Answer:Middle Tennessee, South Carolina, Jacksonville State, Coastal Carolina, St. Peter's, Creighton

> Finished chain.

("Middle Tennessee, South Carolina, Jacksonville State, Coastal Carolina, St. Peter's, Creighton",

 "SELECT DISTINCT mbb_teams.market FROM mbb_teams JOIN mascots ON mbb_teams.market = mascots.market WHERE mascots.mascot IN ('Peacock', 'Rooster', 'Blue Jay', 'Winged Blue Horse')")

It works! We returned the team names that have a mascot that can fly.

The last piece that we hypothetically need is the ability to recognize an ‘advanced’ SQL query that needs to be preprocessed before generating the SQL query.

Function

Here’s a rudimentary sample on how this can be achieved with RegEx.

# @title Request routing logic- Decide to hand to SQL-LLM or enhanced function for pre-analysis before SQL query

import re

def basic_llm_bq():

  print("basic_bq_search")

def enhanced_mascot_bq():

  print("advanced bq_earch")

def get_all_mascots():

  # This query wouldn't need to use the language model FYI

  print("Retrieving mascots")

def question_routing(question):

  print(question)

  if(re.search(".*[Mm]ascot[s]? that have.* in the.*", question)):

    print("matched1")

    print("Calling: {} function".format("basic_llm_bq()"))

  elif(re.search(".*[Mm]ascot[s]? that contain[s]? .*in the.*", question)):

    print("matched2")

    print("Calling: {} function".format("basic_llm_bq()"))

  elif(re.search(".*[Mm]ascot[s]? that contain[s]?.*", question)):

    print("matched3")

    print("Calling: {} function".format("basic_llm_bq()"))

  elif(re.search(".*[Mm]ascot[s]? that equal[s]? .*in the.*", question)):

    print("matched3")

    print("Calling: {} function".format("basic_llm_bq()"))

  elif(re.search(".*[Mm]ascot[s]? that have.*in the.*", question)):

    print("matched4")

    print("Calling: {} function".format("basic_llm_bq()"))

  elif(re.search(".*[Mm]ascot[s]? that are capable of .*", question)):

    print("matched5")

    print("Calling: {} function".format("enhanced_mascot_bq()"))

  elif(re.search(".*[Mm]ascot[s]? that are able to.*", question)):

    print("matched6")

    print("Calling: {} function".format("enhanced_mascot_bq()"))

  elif(re.search(".*[Mm]ascot[s]? are able to.*", question)):

    print("matched7")

    print("Calling: {} function".format("enhanced_mascot_bq()"))

  elif(re.search(".*[Mm]ascot[s]? is .*", question)):

    print("matched8")

    print("Calling: {} function".format("basic_llm_bq()()"))

  elif(re.search(".*[Mm]ascot[s]? contains.*", question)):

    print("matched9")

    print("Calling: {} function".format("basic_llm_bq()()"))

  elif(re.search(".*[Mm]ascot? .*", question)):

    print("matched10")

    print("Calling: {} function".format("basic_llm_bq()()"))

  # Call enhanced query by default

  else:

    print("enhanced_mascot_bq else")

  return

# Samples

Testing

question = "which mascots are able to fly"

question_routing(question)

print("\n")

question = "which mascots fly"

question_routing(question)

print("\n")

question = "which mascot is named ralph"

question_routing(question)

print("\n")

question = "which mascot name contains fly"

question_routing(question)

print("\n")

question = "which mascots can fly"

question_routing(question)

print("\n")

question = "which mascot is a bulldog"

question_routing(question)

print("\n")

Results

These show that basic or advanced questions can be invoked depending on the pattern of the question.

which mascots are able to fly

matched7

Calling: enhanced_mascot_bq() function

which mascots fly

enhanced_mascot_bq else

which mascot is named ralph

matched8

Calling: basic_llm_bq()() function

which mascot name contains fly

matched10

Calling: basic_llm_bq()() function

which mascots can fly

enhanced_mascot_bq else

which mascot is a bulldog

matched8

Calling: basic_llm_bq()() function

Summarizing the Journey:

Reflecting on our Approach

We’ve all heard that LLMs can hallucinate or give incorrect answers. Many times that can be improved through prompt engineering, prompt tuning, model tuning, or through use of additional tooling. Since we’re at the end of this exercise, I’m reflecting on how LLMs should be used and where they can be positioned to shine. Like many things in our field, there’s not always a simple answer and it involves analyzing trade offs.

In case you didn’t notice. This solution is NOT production ready and I’d highly recommend you NOT take the approach that we used in this blog except for education purposes only. 😅

Here are a few of the reasons that this approach isn’t suitable:

The solution isn’t scalable
It’s not performant enough at scale
It would likely have high costs to low value ratio.(our use case wasn’t very practical)
I think quality will continue to improve quickly but I’m not sure if the answers are reliable enough today.
I’m also thinking more about how one might test for regressions with model changes.
I question if an LLM is the right tool today to judge ‘truthiness’
Tech debt would grow quickly if we’re to build this type of functionality as one offs

Alternatives that I’d Explore

If I were to continue exploring a better solution to this challenge, there are two general categories of solutions that I’d explore.

Familiar Technologies

Data Cleanup and Decorating (metadata)
- Clean the data to ensure all data is fully populated. Standard practices like making sure “None” isn’t the value for fields and removing empty strings which can cause weird behavior
- Add additional metadata columns to tables. If ‘flight’ was important for example, we could figure out how to determine ‘flight capabilities’ one time against our data and then set a true/false value in the DB.
  - Benefits of this approach
    - More Performant
    - More Reliable
    - Cleaner/Simpler
    - Cheaper

Exploring Technologies that are newer for me.

Alternative Models
- Wolfram Alpha- Is a model that’s a computational ‘knowledge engine’
  - Benefits
    - Designed to handle factual information- math, science, history, geography
  - Pros
    - Designed to answer factual queries, which aligns well with our use case
  - Cons
    - Expensive- subscription service
    - Can introduce additional delay
    - Fictitious concepts such as ‘mascots’ could be challenging
Encoding words into Embedding or Vector Databases
- Word embeddings are a way of encoding meaning into a multidimensional representation for later comparisons.
  - Embeddings can be analyzed for differences, similarities, and commonalities between objects.
  - Embeddings/Vectors generally have more than 3 dimensions but the below graphic is a 3-dimensional representation of a word vector to help visualize relationships. One can see that words are plotted through multiple dimensions in relation to others.

Graph Database-
- Graph databases are common for social networks to build and visualize relationships between objects and concepts.
- This could be an alternative to building metadata into our SQL database. We’d still have to figure out how to build the important relationships or visibility that we need into our data to make it queryable.
In our example, we could build out the idea of:
- In our example, we could build out the idea of:

“bird” —> <has a relationship to> —> “Can fly”

whereas

“dog” —> has no relationship to “Can fly”.

In this visual, “Tom Hanks” has a relationship (acted_in) “The Polar Express” whereas Keanu Reeves doesn’t.

LLM Wrap Up

Overview

To wrap things up, on a fundamental level LLMs work by:

Statistically analyzing patterns in data that they’ve been trained on
Predicting what word comes next using statistical methods.

Keep in mind that an LLM may experience challenges when asked to:

Follow a complex list of instructions or patterns that aren’t common
Understand meaning of more complex concepts

If suddenly in a situation where you could be experiencing a limitation of the LLM, consider alternative approaches to format the prompt and also if an LLM is the best tool for the job.

Get Involved!

There’s little room for doubt that LLMs and Generative AI will have a profound impact on our world. There’s still so much we need to learn about this type of intelligence.

Try not to let a lack of experience in any subjects such as statistics, linear algebra, calculus, or even programming discourage you. I’ve personally worked with telephony equipment, telephones, PBX’s, datacenter hardware, firewalls, cloud, and application development technologies to name a few.

LLMs are the perfect opportunity for a shift in your career if you feel inspired and are looking for a change of scenery. LLMs have potential to solve some seriously diverse challenges, they’re easy to experiment with, and they’ll likely be the foundation for some innovative solutions in the near future.

Have fun and reach out if you feel compelled to connect!

Blog Notebook

If you’d like to run and or play hands with the code yourself from this blog, download the notebook from the link below. It’ll require you to have a Google Cloud account with access to Vertex AI. You’ll also need to update the variables in the notebook so they utilize resources in your GCP account.

If you’re new to notebooks, Colab is a great way to start playing. There are also some additional ‘Example’ notebooks Colab to better learn how they work. Download the full notebook from this blog and open it in Colab to begin using it.

https://colab.research.google.com/

Full Notebook Download Here:

https://forms.gle/yLUZufyyfEmRy4bZ7

GenAI- A Hands on Exploration (4 of 4)

Pulling Things Together

Routing Logic

Function Definition

Function

Testing

Results

Summarizing the Journey:

Reflecting on our Approach

Alternatives that I’d Explore

Familiar Technologies

Exploring Technologies that are newer for me.

LLM Wrap Up

Overview

More Questions Generated

My curiosity has me asking myself questions like:

Get Involved!

Blog Notebook

Recent Posts

Categories

Archives

Tags

Follow Me