Generation of Humorous Riddles with Prompt Chaining

This paper focuses on the generaiton of punning riddles with MariTalk following the rules of SECO. We then provide a comparison between the two systems and show that they do not differ significantly in terms of funniness.

Install

To install the project, you can use pipenv.

pipenv install

Or by using pip.

pip install -r requirements.txt

Run

The entire project is in the SECO-LLM/scripts folder. The scripts are as follows:

• generate_llm.py: Provides the humor generation prompt chaining, by using MariTalk with an active API key stored in .env.
• sample_evaluation.py: Random sampling of 100 jokes for evaluation.
• evaluation_interface.py: CMOS evaluation interface created with Gradio.
• visualize_evaluation.py: Compute metrics and images for analysing the evaluation results.
• stat_test.R: Do statistical significance testing.

Results

All results can be found in the SECO-LLM/results folder. The folder is organized as follows:

• evaluation/evaluation_2024_04_16_15_55_49.jsonl: Contains all evaluation results. Each evaluator has a single uuid and evaluates joke pairs with unique joke_id. The score sign is already corrected (negative values always represent a preference for SECO).
• img/: Images created for the paper.
• jokes.jsonl: All results from the LLM.Contains the input concept, its sub-words, relations used, all prompts and responses, and the final LLM joke in column joke.
• evaluation_sample.jsonl: Sample of 100 jokes used for evaluation.
• evaluation_scores.jsonl: Evaluation result. Contains the LLM joke in column joke, and SECO's original riddle in adivinha. The final score already aggregates all evaluators' opinions.
• stat_test.txt: Significance testing results.

How to cite

@inproceedings{InacioMarcioLimaGoncaloOliveira2024,
  title     = {Generation of {{Punning Riddles}} in {{Portuguese}} with {{Prompt Chaining}}},
  booktitle = {Proceedings of the 15th {{International Conference}} on {{Computational Creativity}} ({{ICCC}}'24)},
  author    = {In{\'a}cio, Marcio Lima, Marcio and Gon{\c c}alo Oliveira, Hugo},
  year      = {2024},
  month     = jun,
  publisher = {Association for Computational Creativity (ACC)},
  address   = {J{\"o}nk{\"o}ping},
  url       = {https://computationalcreativity.net/iccc24/full-papers/ICCC24_paper_143.pdf}
}

Inácio, Marcio Lima, M., & Gonçalo Oliveira, H. (2024, June). Generation of Punning Riddles in Portuguese with Prompt Chaining. Proceedings of the 15th International Conference on Computational Creativity (ICCC’24). 15th International Conference on Computational Creativity (ICCC’24), Jönköping. https://computationalcreativity.net/iccc24/full-papers/ICCC24_paper_143.pdf