Feature/support better sceanrio goal generation (#92)

* support inference on the whole dataset

* add initial code for scenario and social goal  generation

* modify readme

* add step1 and step2 but still not correct

* add a test

* support generating env and match it with existing agents to be a combo

* add readme

* support gpt-4-turbo change and add db doc and conert to json

* fix prompt to generate better scenario based on gpt-4-turbo

* complete the overall 4 steps

* modify readme

* support bert score filtering

* change name

* delete file

* delete jsonl

* delete dump.rdb

* modify readme

(cherry picked from commit f9bed39)
Signed-off-by: Haofei Yu <1125027232@qq.com>

README.md

@@ -2,9 +2,10 @@
 
 We split our overall framework into multiple parts
 
+0. Data Generate --> Input None / Output new data on redis
 1. Data Processing --> Output general form of sotopia train and test data
 2. Together AI Finetuning --> Input the train and test data / Output model checkpoint
 3. LLM Finetuning --> Input the train and test data / Output model checkpoint
 4. LLM Deplyment --> Input LLM Finetuned model checkpoint / Output Deployable OpenAI type API
 5. Eval --> Input model checkpoint / Output evaluation scores
-6. Generate --> Input None / Output new data on redis
+

data_generate/README.md

@@ -0,0 +1,20 @@
+# Data Generation
+
+For the first step, we generate envProfile (including scenario / social goal / relationship restriction) based on inspiring prompt.
+
+For the 2.1 step, we put the original agentProfile and relationshipProfile into our new redis database
+
+For the 2.2 step, we combine them together to be combos based on conditiona sampling (the restriction is the relationship)
+
+All the EnvProfile (new generated), AgentProfile (sotopia original), RelationshipProfile (sotopia original), and envagentcombo are on the redis database that is new created.
+
+For the third step, we need to use another version of redis and convert it into json file and save the whole data in the database on the local machine.
+
+For the final step, we convert the whole thing into Ruiyi's format.
+
+# Local Redis Setting
+Since the redis-server cannot directly input json data, it requires loading a RedisJson model into the redis-server to enable this function. Therefore, we need to load a docker based on RedisJson:
+
+docker run -p 6379:6379 --name redis-stack redis/redis-stack:latest
+
+Link: <https://github.com/RedisJSON/RedisJSON>