Roleplay Prompt Testing with Promptfoo

• Usage

  • Test different versions of system prompts, or entirely different system prompts at the same time!

  • Test different LLMs, or different temperatures at the same time!

• Prerequisites

  1. Install Promptfoo

     https://www.promptfoo.dev/docs/installation/

  2. Install git (for bash)

     https://git-scm.com/install/

  3. Install VSCode (optional but recommended)
     https://code.visualstudio.com/

  4. Download the roleplay-tester project
     https://github.com/ritodosa/roleplay-tester

• Check out & use the roleplay-tester project

  • The roleplay-tester project has all the sample files it needs, you may test run it right after download!

  • To change it to your own, replace the following contents with your own

    • JohnDoe.md: Your character profile you want to test

    • MarySue.md: Your persona

    • prompt_A.md: Your system prompt you want to test

    • prompt_B.md: A secondary system prompt you want to test alongside prompt_A

    • (full_prompt_A.md, full_prompt_B.md: No need to edited)

    • test_cases/campsite_trip/first_msg.txt: Your first message

    • test_cases/campsite_trip/reply_questioning_the_spray.txt: Your reply

    • test_cases/campsite_trip/reply_spider_threat.txt: A secondary reply

  • Setup the main configuration in promptfoo_config.yaml

    • providers

      • Setup the llms you want in providers. Check this list for supported providers:
        https://www.promptfoo.dev/docs/providers/

    • prompts

      • If you have only one prompt to test, just delete or comment out the second item for full_prompt_B.

    • defaultTest

      • Replace char, user with your persona names. (It is needed so the placeholders in the system prompts can be correctly replaced.)

      • Set the correct filepath for char_persona, user_persona if you created new files other than JohnDoe.md, MarySue.md.

      • You may leave prompt_A, prompt_B as is, unless you renamed it.

    • tests

      • Rename the description, set the correct filepath for first_msg, user_input to whatever you’re using.

  • Running the tests

    1. Open git bash in the folder where promptfoo_config.yaml is located.

    2. Setup you api keys in the git bash (needs to be set everytime you re-open git bash).

        export ANTHROPIC_API_KEY=your_key
        export GEMINI_API_KEY=your_key
        export OPENAI_API_KEY=your_key
        export OPENROUTER_API_KEY=your_key
       

       

    3. Run promptfoo eval

        promptfoo eval -c promptfoo_config.yaml --no-cache
       

       

    4. View the results using

        promptfoo view
       

       

Sample results from 2 different prompts x 2 different LLMs x 2 replies: