Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Getting literal schema responses since Genkit 0.9.x and corresponding plugin version 0.11.0 #204

Open
1 of 7 tasks
TAxdorff opened this issue Jan 8, 2025 · 0 comments
Open
1 of 7 tasks
Labels
bug Something isn't working

Comments

@TAxdorff
Copy link

TAxdorff commented Jan 8, 2025

Affected plugin(s)

  • Anthropic
  • Azure OpenAI
  • Cohere
  • Convex
  • Groq
  • Mistral
  • OpenAI

Describe the bug
Inconsistently (happens often, but never consistently) get literal Schema format responses from generate calls:

Provided data:

{
  "type": "object",
  "properties": {
    "plaintextFullText": "Diet 15 FL OZ 222 ml\n\n",
    "markdownFullText": "# Diet Coke\n\n**Volume:** 15 FL OZ (222 ml)\n\n"
  },
  "required": [
    "plaintextFullText",
    "markdownFullText"
  ],
  "additionalProperties": true,
  "$schema": "http://json-schema.org/draft-07/schema#"
}

Instead the response should have just been the fields within 'properties'.

Required JSON schema:

{
  "type": "object",
  "properties": {
    "plaintextFullText": {
      "type": "string",
      "description": "Full plaintext document ... special characters."
    },
    "markdownFullText": {
      "type": "string",
      "description": "Full markdown document ... markdown formatting."
    }
  },
  "required": [
    "plaintextFullText",
    "markdownFullText"
  ],
  "additionalProperties": true,
  "$schema": "http://json-schema.org/draft-07/schema#"
}

To Reproduce
Send requests using zod schema with this generate statement:

            const generate_response = await ai.generate({
                model: model,
                system: requestDetails.system_prompt,
                prompt: [...textPrompt, ...mediaPrompts],
                config: config,
                output: {schema: outputSchema, format: "json"}
            });

            const data = generate_response.output;

This is shared between calls to gemini and openAI, and worked consistently in older plugin and genkit versions (0.5.17 genkit + <=0.10.1 plugin)

It still works consistently with gemini as well

Expected behavior
Response should just be

{
   "plaintextFullText": "...",
   "markdownFullText": "...",
}

Screenshots
N/A

Plugin(s) version:
Started occurring during 0.10.1 -> 0.11.0 genkitx-openai plugin transition

Currently
0.12.0 genkitx-openai

Genkit version:
Started occurring during 0.5.17 -> 0.9.4 transition with Genkit

Currently using 0.9.12

Additional context
Mostly using 4o and 4o-mini models

Schema definition passed into variable outputSchema:

export const MarkdownFullTextSchema = z.object({
    plaintextFullText: z.string().describe("Full plaintext document ... escape special characters."),
    markdownFullText: z.string().describe("Full markdown document ... markdown formatting."),
});

export type MarkdownText = z.infer<typeof MarkdownFullTextSchema>;
@TAxdorff TAxdorff added the bug Something isn't working label Jan 8, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant