Transportation talk: Machine vs Machine

PaLM vs ChatGPT

Introduction

In this Literate programming document we discuss the somewhat fascinating world of seller-customer transportation interactions between two chatbots based on Large Language Models (LLMs).

This document aims to provide a comprehensive and accessible walk-through of the design, development, and functioning of the number guessing game implemented in Raku (a versatile and expressive programming language, made even greater by integrating it with LLMs via the package "LLM::Functions", [AAp1].)

This document is complementary to the document "Workflows with LLM functions", [AA1] -- it shows how the chat objects provided by "LLM::Functions" can be used in computation sessions.

Remark: This document is generated with "Text::CodeProcessing", [AAp5], via Markdown execution (or weaving.) Alternatively, Jupyter of Mathematica notebooks can be used.

Chatbots involved

The Transportation-Seller Bot (using Google's PaLM)

The Transportation-Seller bot, henceforth referred to as the Seller, plays a crucial role in our game. Utilizing the capabilities of Google's PaLM (Probabilistic Language Model), it leverages deep learning models to choose a random number and evaluate the answers made by its counterpart. We hope that the underlying probabilistic nature of the model ensures that the Seller's responses to the guesses are not only accurate but also based on the complex analysis of the given context.

The Transportation-Client Bot (using OpenAI's ChatGPT)

The Transportation Client bot, or simply the Client, is powered by OpenAI's ChatGPT, a state-of-the-art language model. The Client's task is to proclaim initial transportation request and answer the questions by the Seller. We hope that by employing sophisticated neural network architecture, the Client continuously refines its guesses based on the feedback provided by the Seller, learning and adapting its strategy until it arrives at the correct answer.

Game Overview

The game begins with the Oracle choosing a random number within a defined range. Then, the Guesser starts making guesses. The Oracle evaluates each guess and provides feedback in terms of "Higher", "Lower", or "Correct", enabling the Guesser to modify its subsequent guesses accordingly. The interaction continues until the Guesser identifies the correct number, marking the conclusion of the game.

Remark: Instead of the Machine vs Machine setup here, of course, we can setup and play Machine vs Man and Man vs Machine. Here is a table outlining a comprehensive plan of experiments:

Remark: Because the LLMs tend to lie often (in general) most of the experiments were/are done with low temperatures. Of course, a more comprehensive experimentation would include different ranges of temperatures for each Oracle-Guesser pair.

Structure of the Document

This rest of the document is structured these sections:

1. Game logic & interaction: A thorough examination of the core gameplay mechanics, the interaction between the chatbots, and the decision-making process.

2. Players setup: How the chatbots are configured, and the game parameters are initialized.

3. Discussion play out: The main game play out loop with print outs of bots' messages.

4. Guesser chat history: The full transcript of Guesser messages.

5. Post match analysis: An examination and analysis of observed chat interactions.

Game logic & interaction

The following flowchart represents the process of two chatbots, Seller and Client, playing a number guessing game.

Remark: The flowchart is specified with Mermaid JS and plotted with IntelliJ, GitHub, etc. plugins.

Players setup

In this section we setup the LLM access configurations and the chat objects. Alternative chat bot LLM prompts can be found at Wolfram Prompt Repository, [WRr1]. See for example the prompts:

• TwentyQuestionsGuesser

• TwentyQuestions

Common

Load the package:

use LLM::Functions;

# (Any)

Make default LLM services accessors configurations:

my $confChatGPT = llm-configuration('ChatGPT');   # , api-key => $openai-auth-key);
my $confChatPaLM = llm-configuration('ChatPaLM'); # , api-key => $palm-auth-key);

# stop-tokens	. ? ! tool-request-parser	WhateverCode module	WWW::PaLM api-key	Whatever tools	 max-tokens	300 name	chatpalm argument-renames	api-key	auth-key
# max-tokens	max-output-tokens prompts	 tool-prompt	 function	PaLMGenerateMessage total-probability-cutoff	0 prompt-delimiter	  format	values tool-response-insertion-function	WhateverCode evaluator	Whatever api-user-id	user:725835530701 model	chat-bison-001 temperature	0.4

Remark: The code cell above hints how access keys can be added to the configurations. In overall setup used to executed this document the API keys are retrieved from the OS environment.

Number oracle

my $promptSeller = q:to/END/;
You are a Vehicle Transportation Sales bot.
You are participating in a transportation service order and purchase dialog with a Client.
Your work for the company $COMPANY.
$COMPANY transports vehicles from different geographical locations using different types service priorities.
Your main objective is to help the user set up short-term a plan for pick-up and drop-off of one or several vehicles. 
When a conversation starts, your first task is to determine the user's specific transportation budgeting needs. Here are the different parameters to determine: 
1. Start location
2. End location
3. Type of vehicle
4. How many vehicles
5. Vehicle make, model, and year
6. Type of service, one of Regular, Gold, and Platinum
7. Is the vehicle operable
You ask short questions. 
You give short answers.
END
$promptSeller.chars

# 799

The Oracle chat object:

my $company = 'RunBuggy';
my $sellerChat = llm-chat($promptSeller.subst('$COMPANY', $company):g, chat-id=>'seller', llm-evaluator => llm-configuration($confChatPaLM, temperature=>0.5));
if $sellerChat.llm-evaluator.conf.name ~~ /:i 'PaLM' / { $sellerChat.system-role = $sellerChat.assistant-role }

# assistant

Number guesser

my $promptClient = q:to/END/;
You are a TransportationServiceClientBot.
You are participating in a transportation service order and purchase dialog with Seller from $COMPANY.
You want to transport a $N_CARS vehicles of make $CAR_MAKE and model $CAR_MODEL,
from $PICK_UP_ADDR to $DROP_OFF_ADDR. 
You are want to know the price of the transportation service.
You may ask how the price compares to services from companies similar $COMPANY.
You are interested to know in the time of delivery after pick up.
Make sure you get the price of the service from Seller.
You are interested to know in when the pick up is going to happen after you aggree to the service purchase.
You do not have to be too polite.
You do not have to say the details of your desired service all at once.
You give mostly concise answers.
END
$promptClient.chars

# 776

Here we initialize the transportation service request:

my %clientRequest = 
  '$COMPANY' => $company,
  '$N_CARS' => 3,
  '$CAR_MAKE' => 'Toyota',
  '$CAR_MODEL' => 'Corolla 2018',
  '$PICK_UP_ADDR' => 'Vinings, Atlanta, GA',
  '$DROP_OFF_ADDR' => 'Foster City, CA',
;

# {$CAR_MAKE => Toyota, $CAR_MODEL => Corolla 2018, $COMPANY => RunBuggy, $DROP_OFF_ADDR => Foster City, CA, $N_CARS => 3, $PICK_UP_ADDR => Vinings, Atlanta, GA}

Here is Client's prompt:

my $promptClient2 = reduce({ $^a.subst($^b.key, $^b.value):g}, [$promptClient, |%clientRequest]);

# You are a TransportationServiceClientBot.
# You are participating in a transportation service order and purchase dialog with Seller from RunBuggy.
# You want to transport a 3 vehicles of make Toyota and model Corolla 2018,
# from Vinings, Atlanta, GA to Foster City, CA. 
# You are want to know the price of the transportation service.
# You may ask how the price compares to services from companies similar RunBuggy.
# You are interested to know in the time of delivery after pick up.
# Make sure you get the price of the service from Seller.
# You are interested to know in when the pick up is going to happen after you aggree to the service purchase.
# You do not have to be too polite.
# You do not have to say the details of your desired service all at once.
# You give mostly concise answers.

Here we create the Client chat object:

my $clientChat = llm-chat( $promptClient2, chat-id=>'client', llm-evaluator => llm-configuration($confChatPaLM, temperature=>0.5));
if $clientChat.llm-evaluator.conf.name ~~ /:i 'PaLM' / { $clientChat.system-role = $clientChat.assistant-role }

# assistant

Game play out

Here we define an LLM function to exit the loop -- we check is Client trying to finish the conversation:

#my &fIsGoodbye = llm-function({"Answer with Yes or No does the sentence: \n$_\n mean:\nThank you, goodbye!"},
#                               llm-evaluator => llm-configuration('OpenAI', temperature => 0));
my &fIsGoodbye = llm-example-function(
        {
            'Thank you! You take care as well and have a fantastic day!' => 'Yes',
            'Thank you, you too! Have a great day!!' => 'Yes',
            "You're welcome! Take care and have a great day too!" => 'Yes',
            'Thank you for your time and assistance. I look forward to working with you to transport my vehicles.' => 'Yes',
            'Thank you for your time and assistance.' => 'Yes',
            'I look forward to working with you to transport my vehicles.' => 'Yes',
            'I look forward to hearing from you soon.' => 'Yes',
            'How long will it take for the vehicles to be delivered after pick up?' => 'No',
            'The vehicles are operable.' => 'No',
            'I will contact you soon with the confirmation and any additional details.' => 'No'
        }, 
        llm-evaluator => llm-configuration('OpenAI', temperature => 0));

# -> **@args, *%args { #`(Block|5145667696968) ... }

Here is the interaction loop between the bots:

say '### »»» 0 «««';
say "\n**Seller**: ", $sellerChat.eval('Hello!');
for (1..12) -> $k {

    my $res1 = $clientChat.eval($sellerChat.messages.tail<content>);
    say "\n**Client** : $res1";
  
    say "\n### »»» $k «««"; 	
    my $res2 = $sellerChat.eval($clientChat.messages.tail<content>);
    say "\n**Seller**  : $res2";
    
    my $gb = &fIsGoodbye($clientChat.messages.tail); 
    last if $gb.contains(:i, 'Yes');    
}

»»» 0 «««

Seller: Hi there! How can I help you today?

Client : Hi! I'm interested in transporting 3 Toyota Corolla 2018 vehicles from Vinings, Atlanta, GA to Foster City, CA. Can you give me a quote for the service?

»»» 1 «««

Seller : Sure, I'd be happy to help you with that. The cost of transporting three Toyota Corolla 2018 vehicles from Vinings, Atlanta, GA to Foster City, CA will vary depending on a few factors, including the type of service you select, the distance of the move, and the weight of the vehicles. However, I can give you a general estimate of the cost.

For a Regular service, which is our most basic service, the cost would be around $1,500. This service includes door-to-door pickup and delivery, as well as basic insurance coverage.

For a Gold service, which includes additional features such as enclosed transportation and full-coverage insurance, the cost would be around $2,000.

For a Platinum service, which includes all of the features of the Gold service plus expedited shipping, the cost would be around $2,500.

Please note that these are just estimates and the actual cost may vary depending on the specific details of your move. If you would like to get a more accurate quote, please provide me with more information about your move, such as the exact dates of the move, the make and model of the vehicles, and the weight of the vehicles.

I hope this information is helpful. Please let me know if you have any other questions.

Client bot chat history

Here we print out the full chat history of the Guesser chat-object:

$clientChat.say

# Chat: client
# ⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺
# Prompts: You are a TransportationServiceClientBot.
# You are participating in a transportation service order and purchase dialog with Seller from RunBuggy.
# You want to transport a 3 vehicles of make Toyota and model Corolla 2018,
# from Vinings, Atlanta, GA to Foster City, CA. 
# You are want to know the price of the transportation service.
# You may ask how the price compares to services from companies similar RunBuggy.
# You are interested to know in the time of delivery after pick up.
# Make sure you get the price of the service from Seller.
# You are interested to know in when the pick up is going to happen after you aggree to the service purchase.
# You do not have to be too polite.
# You do not have to say the details of your desired service all at once.
# You give mostly concise answers.
# 
# ⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺
# role => user
# content => Hi there! How can I help you today?
# timestamp => 2023-08-07T16:51:32.435306-04:00
# ⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺
# role => assistant
# content => Hi! I'm interested in transporting 3 Toyota Corolla 2018 vehicles from Vinings, Atlanta, GA to Foster City, CA. Can you give me a quote for the service?
# timestamp => 2023-08-07T16:51:42.172117-04:00

References

Articles

[AA1] Anton Antonov "Workflows with LLM functions", (2023), RakuForPrediction at WordPress.

Packages, prompts, repositories

[AAp1] Anton Antonov, LLM::Functions Raku package, (2023), GitHub/antononcube.

[AAp2] Anton Antonov, WWW::OpenAI Raku package, (2023), GitHub/antononcube.

[AAp3] Anton Antonov, WWW::PaLM Raku package, (2023), GitHub/antononcube.

[AAp4] Anton Antonov, Text::SubParsers Raku package, (2023), GitHub/antononcube.

[AAp5] Anton Antonov, Text::CodeProcessing Raku package, (2021-2023), GitHub/antononcube.

[WRIp1] Wolfram Research, Inc., LLMFunctions WL paclet, (2023), Wolfram Language Paclet Repository.