Method for prompt optimization — Make use of LLM’s internal knowledge

Recent Large Language Models already contain a lot of general knowledge that is available on the World Wide Web. You can access this knowledge directly by asking the LLM a specific question and you’ll get a specific answer.
For example, if you ask ChatGPT what the worlds tallest building is, you will receive an answer like this:

ChatGPT: As of my last update in January 2022, the world’s tallest building is the Burj Khalifa located in Dubai, United Arab Emirates. The Burj Khalifa stands at a height of 828 meters (2,717 feet) and has held this record since its completion in 2010.

However, for more specific questions or questions about something that happened after the training data was aggregated, the response may look like this:

Question: What was the biggest event in Schwäbisch Hall in 2023?
ChatGPT: I apologize, but as of my last update in January 2022, I do not have specific information about events that occurred in Schwäbisch Hall in 2023. To find out about significant events or happenings in Schwäbisch Hall for 2023, I recommend checking local news sources, official event calendars, or the town’s official website for the most current information.

If ChatGPT or any other LLM chat applications encounter such cases, they may either hallucinate or block the request. Therefore, we are standing at the same point as when we had no AI applications: We have to do a web research with search engines like Google or Bing.

But why not combine the power of LLM AI models with classic web research?
Well, that’s not that easy because the world wide web is not optimized for machine processing, it is intended for human use! Websites can be difficult for machines to process due to their complexity, excessive advertising, animations, and a lack of clear separation between relevant and irrelevant content.

So, automating web research can be divided in four steps:

Simplified automated web research process

1. Prompt Revision: Understand the human input and generate an optimized prompt for the requested task (e.g. question answering, research, recommendations, …)
2. Gathering Information: Open the websites, extract relevant content and optimize it for machine processing
3. Filtering Information: Interpretation of the human input and decide what content of step 2 is the most relevant information to answer the question
4. Reporting: Putting together all relevant information fragments and give the user a summarized output with all source URLs

The first step is crucial because the quality of the answer depends mostly on the quality of input context and just second on the content what the LLM creates.

In general, this method can be applied to various tasks. The task requested by the user (examples can be found in step 1) must determined first — To keep it simple, let’s assume a web search scenario and take a closer look at it.

Let’s take the knowledge in the LLM about the task itself to optimize the user input prompt by decoupling the direct user input from the main inference pipeline. In this approach, I call it “Self-induced Prompt Optimization”, the LLM’s knowledge generates task-specific optimization instructions for the user input. With this instructions passed as context system instructions and other static instructions to the LLM, a new prompt based on the user input will be generated.

Process overview of SPO by instruction

In pipeline 1, prompting for optimization instruction creation in this case is quite easy: Just ask the LLM what makes a good web search query!

Prompt: Characterize a good formulated websearch phrase with 10 most important points. Do not consider technical-related things like filters. Focus on the formulation of the phrase itself and not features of the websearch engine.
GPT 3.5:

Clarity: The phrase should be clear and concise, avoiding ambiguity. Use straightforward language that clearly conveys what you are looking for.

Specificity: Be as specific as possible about the topic or information you’re seeking. General queries often yield broad results, while specificity narrows down the focus.

Relevance: Ensure that the keywords or terms you include are directly related to your query. Irrelevant terms can lead to off-topic results.

Use of Keywords: Incorporate relevant keywords or key phrases that are likely to appear in the content you are seeking. Think about the terms someone might use when discussing your topic.

[…]

After slight re-formatting, this output can be inserted into a prompt template and passed to the API for inference.

Prompt context with web search specific optimization instructions

This example is just a standalone example for web search. If implemented into an autonomous web search agent, it would be useful to formulate the task definition more open with some parameters and add instructions to format the output as JSON for better handling.

In the following, I will give you some examples for using this method to optimize search phrases for various tasks.

Some examples while testing on command line

Examples for classic question answering

These are examples for everyday problems which can be solved by a web search:

Question: The Apple Watch is not visible on my new iPhone and I cannot connect it. How do I configure it for the new phone?
Generated search queries:
1: pair Apple Watch with new iPhone
2: Apple Watch not showing up on iPhone troubleshooting
3: configure Apple Watch for different iPhone

Question: My coffee machine has stopped working, no water comes out. Maybe it needs cleaning? But I don’t know how to do that.
Generated search queries:
1: coffee machine not dispensing water troubleshooting
2: how to clean a coffee maker
3: coffee machine no water output causes

Examples for explicit information request

These are examples for searching facts or time dependent information.

Question: What’s the worlds tallest building in 2023?
Generated search queries:
1: tallest building in the world 2023
2: current world’s highest skyscraper 2023
3: list of tallest buildings 2023

Question: When will the Red Hot Chili Peppers play their next concert in an european city?
Generated search queries:
1: Red Hot Chili Peppers next concert date European city
2: Upcoming Red Hot Chili Peppers shows Europe
3: Red Hot Chili Peppers tour schedule Europe

Examples of web research and investigation

Sometimes a web search is more than just answering a single question, but rather a collection of information.

Question: Make a research about the CO2 footprint of battery electric vehicles and compare with an equivalent vehicle with a modern Diesel engine.
Generated search queries:
1: CO2 footprint comparison battery electric vs modern diesel vehicles
2: Lifecycle emissions of electric vs diesel cars
3: Environmental impact battery electric vehicles vs diesel engines

Question: List me the three most important pros and cons of artificial intelligence.
Generated search queries:
1: pros and cons of artificial intelligence
2: benefits and drawbacks of AI technology
3: AI advantages disadvantages summary

Question: Is it true that an apple a day keeps the doctor away? Give me the reason for the answer.
Generated search queries:
1: Does an apple a day keep the doctor away reason
2: Health benefits of eating apples daily
3: Myth or fact an apple a day medical perspective

Finally, check whether the first search phrase of the last example returns good results:

The results of a search with one of the search phrases

I think that looks quite good! In this case, the first three pages would probably answer the question.

Another example: Process description generation

Generating a stepwise process description based on a textual description of a software process can be another possible field of application for this method.
The user inputs a process formulated in natural language and the LLM transforms it into a stepwise process description using language-independent pseudocode for each step. The LLM will use its own knowledge about how a good pseudocode description should look like.

Process description by the user:
The program for evaluating fuel prices opens a textfile and checks, if there is correct formatted data in CSV format. Then, it puts all the data into an array and converts the column with the name “Total amount” to integers and the one with the name “Current price” to float. Then multiply “Current price” with “Total amount” in all rows with the value “Gasoline” in column “Type”.

Process description output with self-induced prompt optimization context for this task:

# Process Title: Evaluate Fuel Prices
# Objective: Process a CSV file to calculate the total cost for gasoline entries.

# Input: Text file containing fuel data in CSV format.
# Output: Array with calculated total costs for gasoline entries.

# Initialize variables
Open the text file for reading
Initialize an empty array for storing processed data

# Step 1: Validate CSV Data Format
# Check if the data in the text file is in correct CSV format
Read the first line of the file and check for correct headers

# Step 2: Read and Store Data
# Read each line from the file, split by commas, and store it in an array
While not end of file:
    Read line and split by comma to form a list
    Append list to array

# Step 3: Convert Data Types
# Convert "Total amount" to integer and "Current price" to float for each entry
For each row in array:
    Convert row["Total amount"] to integer
    Convert row["Current price"] to float

# Step 4: Calculate Total Cost for Gasoline Entries
# Multiply "Current price" with "Total amount" where "Type" is "Gasoline"
For each row in array:
    If row["Type"] equals "Gasoline":
        Multiply row["Current price"] by row["Total amount"]
        Store result back into array or a new result structure

# Termination or Finalization
Close the text file
Return or output the modified array with calculated costs

If we let ChatGPT directly formulate the pseudocode without instruction context, we’ll get this:

1. Open the text file containing fuel prices data.
2. Read the content of the text file line by line.
3. Check if the data is correctly formatted in CSV format:
   IF Correctly formatted THEN
      4. Initialize an empty array named 'fuelDataArray'.
      5. FOR each line in the file:
         6. Split the line by commas to extract individual data fields.
         7. Create a dictionary or a similar data structure to hold the data of each row.
         8. Append the dictionary to the 'fuelDataArray'.
      
      9. FOR each item in 'fuelDataArray':
         10. IF 'Type' column of the item is "Gasoline" THEN
             11. Convert the "Total amount" column of the item to an integer.
             12. Convert the "Current price" column of the item to a float.
             13. Multiply the values of "Current price" and "Total amount" columns.
      
      14. Output or store the result after multiplication as required.
   ELSE
      15. Display an error message indicating incorrect format.
END IF

That’s not bad but very chaotic and not well commented and formatted.
And it is not 100% correct, see step 10: The data types have been already converted before, this is not necessary in this step.

Conclusion

This method can be applied to various tasks. Especially its use in an automated web research pipeline is quite interesting.
Since all information in LLM becomes outdated as soon as the training starts, it is logical to use only general knowledge in LLM, especially in understanding the meaning of human input.
The main goal is always to make human content/input understandable for machines, which means converting it to a generalized form.

Code examples

See Github repo for this article:
https://github.com/jschw/advanced-rag-websearch/tree/main/examples/1_prompt_optimization