How expand Kernel Memory Output #9877

peopleworks · 2024-12-04T15:28:36Z

peopleworks
Dec 4, 2024

Hi

Everything is working fine, I get the right answer, fast response, but the answer is cut off at one point and does not give me completeness even though the part it presents is correct and in the order that I requested it.

`
public async Task AskInsightsAsync()
{

var question = @"
Data Insights Extraction Prompt

Objective: Perform an in-depth analysis of the provided JSON file, uncovering meaningful insights, patterns, and actionable intelligence.

Data Exploration Steps:

1. Visualization Recommendations
- Recommend the most appropriate visualization types for key insights
- Using Mermaid diagrams suggest potential charts, graphs, or diagrams that would effectively communicate the data's story. Ensure the diagram is clear and concise, avoiding over-complication or information overload

2. Comprehensive Data Profiling
- Calculate and report:
  * Total number of records/entries
  * Distribution of data across different keys
  * Unique value counts for categorical variables
  * Statistical summary for numerical variables (mean, median, min, max, standard deviation)

3. Key Insights Extraction
- Identify the most significant patterns and trends in the data
- Highlight:
  * Top N values for important categorical variables
  * Unusual or outlier data points
  * Correlations between different attributes
  * Potential hidden relationships in the data

4. Advanced Analysis
- Perform contextual analysis based on the nature of the data
- Develop hypotheses about potential underlying patterns
- Suggest potential business or research implications of the findings


5. Data Quality Assessment
- Evaluate data completeness
- Identify missing values or inconsistencies
- Assess the reliability and potential biases in the dataset

Deliverables:
- Comprehensive written report of findings
- Summary of key insights
- Recommendations for further investigation
- Potential data-driven action items

Additional Instructions:
- Provide clear, concise explanations
- Use technical yet accessible language
- Back up observations with specific data references
- Highlight both obvious and non-obvious insights

Formatting Preferences:
- Always respond in proper HTML format, but do not include <html>, <head>, or <body> tags.
- Include headers and subheaders
- Utilize bullet points and numbered lists for readability
- Provide code snippets or pseudo-code if relevant

Please process the JSON file with these guidelines, demonstrating deep analytical thinking and attention to detail.
";

var answer = await memoryConnector.AskAsync(question, filter: MemoryFilters.ByDocument("json01"));

return answer.Result;

}
`

Thank you in advance

Answered by sphenry

Dec 17, 2024

@peopleworks it might be worth trying a run with gpt-4o (not mini) to see if the longer context window helps. The Kernel Memory discussion board is here, and they might be able to help as well: https://github.com/microsoft/kernel-memory/discussions

View full answer

moonbox3 · 2024-12-08T23:42:13Z

moonbox3
Dec 8, 2024
Collaborator

Tagging @markwallace-microsoft for visibility.

0 replies

sphenry · 2024-12-09T16:43:19Z

sphenry
Dec 9, 2024
Collaborator

@peopleworks a couple questions:

Are you using the Kernel Memory project https://github.com/microsoft/kernel-memory, or are you using Memory with Semantic Kernel?
On the surface this feels like a there may have been a limit on the number of tokens returned, can you look at the FinishReason of the Content?

1 reply

peopleworks Dec 9, 2024
Author

Hi @sphenry

I'm using Kernel Memory project , and I think this is the problem, because for example : if I use it to get Text2SQL using a document with database schema, works perfect, because the result is short. My big problem, I don't know how do this without Kernel Memory, just using Semantic Kernel. that's why I asking how to expand the output.

Please, if you have another way to handle this let me know, I'm exploring any ideas

Note: I going to share my appsetting.json here, maybe some bad parameters. By the way I'm using gpt-4o-mini as Text Model.

Thank you so much for your time and response
appsettings.json

peopleworks · 2024-12-10T04:03:25Z

peopleworks
Dec 10, 2024
Author

Hi @sphenry

Tested is a Kernel Memory problem, I use only Semantic Kernel + ITextEmbeddingGenerationService + tokenizer and works perfect

2024-12-09_22-52-18.SKTK.mp4

Maybe another better way that I can persist the file and use from a Database.

Thank you
Pedro

0 replies

sphenry · 2024-12-17T16:49:59Z

sphenry
Dec 17, 2024
Collaborator

@peopleworks it might be worth trying a run with gpt-4o (not mini) to see if the longer context window helps. The Kernel Memory discussion board is here, and they might be able to help as well: https://github.com/microsoft/kernel-memory/discussions

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

How expand Kernel Memory Output #9877

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 4 comments 1 reply

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

How expand Kernel Memory Output #9877

peopleworks Dec 4, 2024

Replies: 4 comments · 1 reply

moonbox3 Dec 8, 2024 Collaborator

sphenry Dec 9, 2024 Collaborator

peopleworks Dec 9, 2024 Author

peopleworks Dec 10, 2024 Author

sphenry Dec 17, 2024 Collaborator

peopleworks
Dec 4, 2024

Replies: 4 comments 1 reply

moonbox3
Dec 8, 2024
Collaborator

sphenry
Dec 9, 2024
Collaborator

peopleworks Dec 9, 2024
Author

peopleworks
Dec 10, 2024
Author

sphenry
Dec 17, 2024
Collaborator