Frequent error I'm getting: "Could not format the output of the model into a dictionary or a Pydantic model"

[aalyousfi]

Hello,

I'm getting this error frequenlty with Outlines:

TypeError: Could not format the output of the model into a dictionary or a Pydantic model. The model has likely exceeded its context length. Please try again using `constr` (for Pydantic) and `maxLength` (for JSON Schema) to limit the length of the string fields. If this exception is raised nevertheless please open an issue: https://github.com/outlines-dev/outlines/issues

I specified maximum length for all items in my schema and I set the max generated tokens to something like 2048, so I don't think the context length is the problem. And I've encoutered the error with many models, both transformers and AWQ. And I don't think resources is an issue because I've ~90GB VRAM and ~180GB RAM.
And the most frustrating part is that the error is unpredictable, sometimes it happens, sometimes not. I couldn't determine what causes it to happen But I provide below an example code that just produced the error.
Please help me with this issue as it's the main problem I'm facing with Outlines and it's preventing me from having a robust system.

Here is the full error trace:

---------------------------------------------------------------------------
JSONDecodeError                           Traceback (most recent call last)
File ~/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages/outlines/generate/api.py:236, in SequenceGenerator.__call__(self, prompts, max_tokens, stop_at, rng, kv_cache)
    235 try:
--> 236     formatted = [self.format_sequence(sequence) for sequence in stripped]
    237 except pyjson.decoder.JSONDecodeError:

File ~/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages/outlines/generate/api.py:236, in <listcomp>(.0)
    235 try:
--> 236     formatted = [self.format_sequence(sequence) for sequence in stripped]
    237 except pyjson.decoder.JSONDecodeError:

File ~/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages/outlines/generate/api.py:435, in json.<locals>.<lambda>(x)
    434     generator = regex(model, regex_str, max_tokens, sampler)
--> 435     generator.format_sequence = lambda x: pyjson.loads(x)
    436 else:

File ~/anaconda3/envs/pytorch_p310/lib/python3.10/json/__init__.py:346, in loads(s, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw)
    343 if (cls is None and object_hook is None and
    344         parse_int is None and parse_float is None and
    345         parse_constant is None and object_pairs_hook is None and not kw):
--> 346     return _default_decoder.decode(s)
    347 if cls is None:

File ~/anaconda3/envs/pytorch_p310/lib/python3.10/json/decoder.py:337, in JSONDecoder.decode(self, s, _w)
    333 """Return the Python representation of ``s`` (a ``str`` instance
    334 containing a JSON document).
    335 
    336 """
--> 337 obj, end = self.raw_decode(s, idx=_w(s, 0).end())
    338 end = _w(s, end).end()

File ~/anaconda3/envs/pytorch_p310/lib/python3.10/json/decoder.py:353, in JSONDecoder.raw_decode(self, s, idx)
    352 try:
--> 353     obj, end = self.scan_once(s, idx)
    354 except StopIteration as err:

JSONDecodeError: Expecting ',' delimiter: line 976 column 1 (char 7784)

During handling of the above exception, another exception occurred:

TypeError                                 Traceback (most recent call last)
Cell In[3], line 116
    112 print(t)
    113 prompt = classification_prompt(t, categories, examples, sys_prefix=f"<|im_start|>system\n", 
    114                                user_prefix="<|im_end|>\n<|im_start|>user\n", 
    115                                user_suffix="<|im_end|>\n<|im_start|>assistant")
--> 116 clas = generator(prompt, max_tokens=2048)

File ~/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages/outlines/generate/api.py:238, in SequenceGenerator.__call__(self, prompts, max_tokens, stop_at, rng, kv_cache)
    236     formatted = [self.format_sequence(sequence) for sequence in stripped]
    237 except pyjson.decoder.JSONDecodeError:
--> 238     raise TypeError(
    239         "Could not format the output of the model into a dictionary or a Pydantic model."
    240         + " The model has likely exceeded its context length. Please try again using `constr` (for Pydantic)"
    241         + " and `maxLength` (for JSON Schema) to limit the length of the string fields. If this exception"
    242         + " is raised nevertheless please open an issue: https://github.com/outlines-dev/outlines/issues"
    243     )
    245 return formatted if len(formatted) > 1 else formatted[0]

TypeError: Could not format the output of the model into a dictionary or a Pydantic model. The model has likely exceeded its context length. Please try again using `constr` (for Pydantic) and `maxLength` (for JSON Schema) to limit the length of the string fields. If this exception is raised nevertheless please open an issue: https://github.com/outlines-dev/outlines/issues

And here is an example code that produced this error:

categories = "Auto & Vehicles\nBeauty & Fitness\nBooks & Literature\nBusiness & Industrial\nComputers & Electronics\nFinance\nFood & Drink\nGames\nHealth\nHobbies & Leisure\nHome & Garden\nInternet & Telecom\nJobs & Education\nLaw & Government\nNews\nOnline Communities\nPeople & Society\nPets & Animals\nReal Estate\nReference\nScience\nSensitive Subjects\nShopping\nSports\nTravel"
with open("ts_categories.txt", encoding='utf-8') as cf:
    categories = cf.read()
categories = re.sub(r"^\- ", "", categories, flags=re.MULTILINE)
Category = Enum('Category', {c: c for c in categories.split('\n')})

class Classification(BaseModel):
    # category: str = Field(description="The category assigned to the video")
    reasoning: str = Field(description="The reasoning behind choosing the category", max_length=220)
    category: Category = Field(description="The category assigned to the video")
    fit_score: Decimal = Field(description="A score for how fitting the category is for the video", max_digits=3, decimal_places=2)


class VideoClassifications(BaseModel):
    video_title: str = Field(description="The video title to classify", max_length=100)
    classifications: conlist(Classification, max_length=3) = Field(description="The classification(s) of the video", )
    
with open("ts_categories_h.txt", encoding='utf-8') as cf:
    categories = cf.read()
    
titles = [
    """Kone Elevator at Evolve Concept Mall in Subang, Selangor, Malaysia(Scenic)""",
    """Dank Indian Memes REACTION | Indian Memes | Indian Memes Compilation | Last Night  | Neha M.""",
    """Finish in the top 10 in Livik 12 times in Classic Mode PUBG MOBILE""",
    """This Isn't Hogwarts! A Harry Potter Song""",
    """Dance Moms: Problems with Cathy (S1, E4) | Lifetime""",
    """HIV/AIDS Life Beyond the Diagnosis""",
    """How to Say Numbers 21 thru 30 in German | German Lessons""",
    """How to make Heston Blumenthal's Christmas pudding""",
    """McKim vows action on climate change""",
    """Ollie (formerly called Sphero 2B) Demo and Overview""",
    """Urban Decay, Naked 3 !""",
    """Misha Collins - [PART 1] - Bandaid Auction/Making Fans Uncomfortable -SPN  VanCon 2010""",
    """العلاقات الأمريكية الكوبية...تطبيعٌٌ بعد قطيعة دامت أكثر من نصف قرن""",
    """Raising A Child w/Special Needs - Ages 4-6""",
    """Baby boy gift decorations ideas"""
]


examples = [
    VideoClassifications(
        video_title="Far Cry Map Editor PC",
        classifications=[
            Classification(
                reasoning="The video references 'Far Cry' which is a well-known video game series, and it mentions a specific aspect of gaming (map editing).", # So it can be classified under 'Entertainment > Video Gaming'.",
                category=Category("Entertainment > Video Gaming"),
                fit_score=0.9
            )
        ]
    ),
    
    VideoClassifications(
        video_title="2015 GMC Canyon Pickup Truck Walkaround @ Detroit Auto Show (NAIAS) 2014",
        classifications=[
            Classification(
                reasoning="The title is centered around a specific vehicle model, presented in a format that is informative for potential buyers.", # So it can be classified under 'Wheels & Motors > Shopping & Buying'",
                category=Category("Auto & Motors > Cars & Passenger Trucks"),
                fit_score=0.9
            ),
            Classification(
                reasoning="The video focuses on a popular public entertainment event (the auto show), which is a popular entertainment event where new vehicles are showcased.", # So it can be classified under 'Entertainment > Things to Do & See'",
                category=Category("Entertainment > Things to Do & See"),
                fit_score=0.75
            )
        ]
    )
]


@outlines.prompt
def classification_prompt(v_title: str, available_categories: str, examples: list[VideoClassifications], 
                          sys_prefix="", user_prefix="", user_suffix=""):
    """
{{ sys_prefix }}You are a knowledge worker who is experienced in content classification.
Your job is to classify given video titles into one or more of the categories given below.
First you will reason about the video and what it is talking about.
Then you'll choose one or more suitable categories for the video.
Then you'll assign a fit score to indicate how fitting the category is to the video.



---
Available Categories:
{{ available_categories }}
---



Here are some examples of video titles that you have successfully classified before:

{% for example in examples %}
Video Title: 
{{ example.video_title }}

Classifications:
{{ example.model_dump_json() }}
{% endfor %}

{{ user_prefix }}
Here is the new video title that you need to classify:
Video Title: 
{{ v_title }}

Classifications:{{ user_suffix }}
    """
    
model = models.transformers('PygmalionAI/mythalion-13b', device="auto", model_kwargs={})
generator = outlines.generate.json(model, json.dumps(VideoClassifications.model_json_schema()))
for t in titles:
    print(t)
    prompt = classification_prompt(t, categories, examples, sys_prefix=f"<|im_start|>system\n", 
                                   user_prefix="<|im_end|>\n<|im_start|>user\n", 
                                   user_suffix="<|im_end|>\n<|im_start|>assistant")
    clas = generator(prompt, max_tokens=2048)

[rlouf]

This may be a problem due to tokenization. Could you run the example with pdb, go up a few frames and print the value of the variable generated defined here? I would run the example myself but I don't have a GPU with enough memory to run it locally.

[aalyousfi]

Sure, I'll try to do that in the next few days and report the results here.

[aalyousfi]

Ok, here is the generated value for when the error occurs:

['{"video_title":"Dank Indian Memes REACTION | Indian Memes | Indian Memes Compilation | Last Night | Neha M.","classifications":[{"reasoning":"The content of the video includes memes, which are popular in digital culture. The presence of the words \'Last Night\' and \'Neha M.\' suggests that this could be related to entertainment or social content, possibly from a "\n            \n       \n     \n    \n       \n    \n    \n     \n    \n    \n       \n\n     \n     \n    \n       \n\n     \n     \n    \n        \n      \n    \n     \n     \n    \n       \n\n     \n    \n    \n     \n     \n    \n    \n     \n     \n    \n        \n      \n    \n     \n    \n     \n\n     \n     \n    \n   \n    \n     \n    \n     \n   \n    \n     \n    \n     \n\n     \n    \n    \n     \n     \n\n\n   \n    \n     \n    \n     \n   \n     \n    \n    \n     \n     \n\n    \n\n\n\n\n\n\n\n\n    \n    \n    \n\n   \n     \n    \n   \n   \n   \n     \n    \n     \n    \n    \n     \n     \n    \n    \n     \n     \n \n    \n   \n   \n \n  \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n']

It's weird to see this.. how is it generating this long sequence (thousands of characters) for the 'reasoning' field although I specified its max_length to be 220!

And by the way, the error occurs at different places. For example, this time it occurred at this title: "Dank Indian Memes REACTION | Indian Memes | Indian Memes Compilation | Last Night | Neha M." but previously it occurred with other titles. And sometimes it works with the same titles without errors.

[aalyousfi]

Hi @rlouf . Just wanted to make sure you saw my reply above?

[rlouf]

If max_length is exceeded this is definitely a bug and we should open an issue. As for the fact that it works sometimes and sometimes not is a sampling artifact. We're soon going to introduce other sampling methods that may help with this.