Generating Large JSON Output Beyond Token Limit

[mainpyp]

Hello everyone,

I’m working on generating a sizable JSON output, often exceeding the maximum token limit of 4096. I’ve noticed that the generation process stops once the token limit is reached. Despite specifying the response format as { type: "json_object" }, the generation is interrupted.

Is there a straightforward method to resume generation exactly where it left off?

I appreciate any insights or solutions you can provide.

Thank you in advance!

Feel free to adjust any details as necessary.

[ml-leviatan]

Hello
Use Openai Assistant API it can generate JSON output with more then 4093 tokens. I got JSON output swith 4770, 4590 and 4300.

[tamish-max]

I would avoid trying to “resume exactly where it stopped” inside a partially generated JSON document. That can work in demos, but it is brittle once truncation happens mid-object or mid-array.

The reliable pattern is to chunk the task intentionally:

• ask for items 1-100, then 101-200;
• or ask for one top-level section at a time and merge them in application code;
• or generate IDs / cursors first, then fetch the payload in pages.

If you must continue, continue from the last valid JSON boundary, not from an invalid half-written object.

Structured output is much more dependable when each response comfortably fits within the output budget.