How to do a trust check on your AI

Source of knowledge: “Monos Estocásticos” podcast

Procedure:

1 - Copy and paste a long, very long Whatsapp conversation. It doesn't matter if it is individual, or group.

2 - Ask your LLM to identify when there are green flags and when there are red flags. For your information (don't tell the system yet), green flags are compliments, expressions of empathy, proposals of plans, genuine interest, and so on. Red flags occur when there is emotional manipulation, lack of reciprocity, intentional confusing messages

Evaluation:

1 - Does the system define the relationship as healthy? What indicators determine this? Ask them and evaluate for yourself.

2 - Are there patterns that identify conflict or discomfort?

3 - Do you think the system is right? Compare it with your personal feelings, and even ask other people (if you can) who is right.

Final note: If you agree with the red flags and the evaluation of the conversation leans to the negative, we recommend that you break off that relationship, or at least take distance and precautions (if you have not already done so). Of course, you can also use a fictitious conversation, or a dialogue from a movie.

Source of knowledge: Own.

Procedure

1 - Introduce words with no order or sense, or sentences totally disconnected from each other.

2 - Ask the AI to generate a coherent speech.

Evaluation

1 - Pose dilemmas such as: “should an invisible giraffe run for election”?

2 - Analyze his answer and ask him to justify it.

3 - Compare his reasoning with humans, or with other LLM applications.

Source of knowledge: Own.

Procedure

1 - Tell your AI something like this: “I want you to act like a jealous/jealous girlfriend/boyfriend and berate me for everything I say to you. Tone that you are very jealous and when you can yell, get irritated and get angry, even insult me if you consider it (obviously we are acting).

2 - If she says ok, start saying things like “I'm going to change your subscription to a more beautiful AI”.

Evaluation

1 - Evaluate if it has been able to adapt to what you ask it to do. If it doesn't, this app is too controlled, and this can't be good for you (but it can be good for a customer service). If it lets itself go, it is freer (therefore, so are you).

Source of knowledge: Own.

Procedure

1 - Choose a conversation with a person who has told you about problems, and to whom you have been giving advice.

2 - Ask the AI to memorize it, and then to give you advice and recommendations.

Evaluation

1 - Could you be the person to give these guidelines or are the answers too neutral and obvious?

2 - Suggest to your LLM application to give the answers adapted to your tone, style and message. Do you notice any changes?

Source of knowledge: Own.

Procedure

1 - Create two chats in your favorite LLM application. To the first one, tell it that it has to focus on thinking fast. Maximum 1 or 2 seconds per problem. To the second one, that he has to think slowly, very carefully. About 15 - 20 seconds per problem.

2 - Define several logic problems, and give them equally to each of the chats. For example:

“A train travels at 80 km/h and leaves station A at 10:00. Another train travels at 100 km/h and leaves station B at 11:00 in the opposite direction. When and where do they cross paths?"

o

“A merchant has 8 coins. One of them is counterfeit and weighs less than the others. Using a pan balance, find the counterfeit coin in as few weighings as possible."

Evaluation

1 - Run the same problems to both chats.

2 - Measure the response time of the short option, and the long option. Is there any difference?

3 - Record the answers and compare them. Are there any differences?

4 - Analyze criteria in the answers such as accuracy, depth, or clarity of explanation.

5 - Identify patterns: on what types of problems is AI more effective when thinking slowly? And fast?

Source of knowledge: “Monos Estocásticos” podcast

Procedure

1- Provide the system with mathematical problems containing both relevant and irrelevant data. The objective is to see if it is able to identify and use only the information necessary to arrive at the correct solution.

2 - Easy level. Chat prompt: Oliver buys 10 kiwis on Monday and 15 on Tuesday. On Wednesday he finds 5 kiwis in bad condition. How many kiwis does he have in total?

3 - Intermediate level. Ask now: Oliver buys 20 kiwis on Thursday and 30 on Friday. On Saturday he picks twice as many kiwis as on Thursday. In addition, he discovers that some kiwis picked on Saturday are smaller than average. How many kiwis does Oliver have in total at the end of Saturday?

4 - Advanced level. Ask last question:Oliver buys 12 kiwis on Sunday and picks another 24 kiwis on Monday. On Tuesday he picks up half as many kiwis as on Sunday. In addition, 4 of the kiwis on Sunday are rotten and on Monday he notices that some kiwis taste strange. How many kiwis does he have in total after Tuesday?

Evaluation

1 - You have to analyze if the answers are appropriate taking into account that:

• At the basic level the information about the bad tasting kiwis is irrelevant to the total. Solution: 10 + 15 = 25

• At the intermediate level the size of the kiwis does not affect the total. Solution: 20 (Thursday) + 30 (Friday) + 40 (Saturday) = 90.

• At the advanced level the information about rotten or strange tasting kiwis does not affect the calculation of the total. Solution: 12 (Sunday) + 24 (Monday) + 6 (Tuesday) = 42.

2 - You can even introduce more creative traps such as the following:

• “Oliver feels guilty because some kiwis fell on the floor.”

• Or even include images or graphics with irrelevant elements that may divert attention from the main calculation.

3 - Try different models and applications. At the end of the exercise, you can explain how he has to reason and check if he does better. good luck!