Basic KuzuGraph Example fails when using Ollama vs OpenAI

[wingraver]

KuzuDB is great for a lot of use cases and I'm trying to expand usage scenarios. The knowledge graph example presented by both Kuzu and LangChain consistently fails at this step:

Add the graph document to the graph
graph.add_graph_documents(graph_documents, include_source=True,)

Results in...
RuntimeError: Binder exception: Cannot find property id for e.

It would be great to be able to use Ollama and return comparable results; as of now though I'm not seeing a way to resolve this... no local model has been able to get pass this point (and I have many)???

Additionally, there appears to be another issue as the generated nodes and (missing) relationships are usually incomplete; relative to the results presented in the example. In a lot of cases no relationships are returning when the schema is strictly applied? There should be a way to govern what the LLM returns for addition to the knowledge graph...

[wingraver]

The missing relationships after converting graph documents appear to result from the strict_mode = True totally ignoring or being unable to resolve for the provided relationship schema?

[ray6080]

hi @wingraver thanks for reporting this! Will look into it on our side.

[prrao87]

Hi @wingraver, I think the underlying issue here is that the open source LLM in Ollama is smaller and less capable than the commercial LLMs like OpenAI's or other providers. As a result, the LLMGraphTransformer in LangChain potentially isn't capturing the nodes and relationships correctly. Could you let me know which exact model you're using in Ollama so I can run some experiments on my end? Thanks!

[wingraver]

Hi @prrao87 thanks for the feedback. I don't believe it's the open source LLM... I've used a number of them: llama3.1:8b; phi4-mini; qwen3:32b-q8_0; gemma3:27b-it-qat; deepseek-r1:32b; deepseek-r1:8b (the recent 0825 version); and even devstral.

What is consistently true is that when strict_mode = False, nodes and relationships are generated; their names not matching with those put forward by the schemas used in the demo.

Then when strict_mode = True, some nodes might be generated but no relationships are.

Those open-source models are varied in size, capability, and provider, it would be very doubtful that they all couldn't generate nodes and relationships from a text??? Perhaps the coding instructions of LLMGraphTransformer are directly aligned/tied to OpenAI? Were you able to reproduce the error using the kuzudb-langchain demo?

[prrao87]

That's interesting, perhaps the default system prompt is more tuned to OpenAI's models than others. I'll do some more experiments with these models you listed and get back to you.

[prrao87]

Hi @wingraver, I was looking at reproducing the langchain-kuzu demo and yes, you're right: the default strict_mode=True results in empty relationships when using the Ollama models (I tried llama3.1:8b and phi4-mini). Below are the extracted GraphDocument results for the text "Tim Cook is the CEO of Apple. Apple has its headquarters in California.".

✅ OpenAI gpt-4o-mini with strict_mode=True

The full set of nodes and relationships are captured by the OpenAI gpt-4o-mini model.

[
    GraphDocument(
        nodes=[
            Node(id="Tim Cook", type="Person", properties={}),
            Node(id="Apple", type="Company", properties={}),
            Node(id="California", type="Location", properties={})
        ],
        relationships=[
            Relationship(source=Node(id="Tim Cook", type="Person", properties={}), target=Node(id="Apple", type="Company", properties={}), type="IS_CEO_OF", properties={}),
            Relationship(source=Node(id="Apple", type="Company", properties={}), target=Node(id="California", type="Location", properties={}), type="HAS_HEADQUARTERS_IN", properties={})
        ],
        source=Document(
            metadata={},
            page_content="Tim Cook is the CEO of Apple. Apple has its headquarters in California."
        )
    )
]

❌ Ollama llama3.1:8b with strict_mode=True

No relationships are captured, only nodes are captured (not very useful).

[
    GraphDocument(
        nodes=[
            Node(id='Tim Cook', type='Person', properties={}),
            Node(id='Apple', type='Company', properties={})
        ],
        relationships=[],
        source=Document(
            metadata={},
            page_content='Tim Cook is the CEO of Apple. Apple has its headquarters in California.'
        )
    )
]

❌ Ollama phi4-mini with strict_mode=True

To reveal more, when you inspect the graph documents with strict_mode=False, you can see that the phi4-mini model sometimes gets the relationship's source and target Node IDs correct, for e.g., "Tim Cook" and "Apple". However, for other cases, for "Apple" and "California", the "California" node wasn't extracted, so when we pass this to Kuzu, we're trying to create a relationship to a node that doesn't exist in the GraphDocument data.

GraphDocument(
    nodes=[
        Node(id='Tim Cook', type='Person', properties={}),
        Node(id='Apple', type='Company', properties={})
    ],
    relationships=[
        Relationship(source=Node(id='Tim Cook', type='Person', properties={}), target=Node(id='Apple', type='Company', properties={}), type='CEO', properties={}),
        Relationship(source=Node(id='Apple', type='Company', properties={}), target=Node(id='California', type='Location', properties={}), type='HEADQUARTERS_LOCATION', properties={})
    ],
    source=Document(
        metadata={},
        page_content='Tim Cook is the CEO of Apple. Apple has its headquarters in California.')
    )
]

Takeaways

The root cause of the issue is that the open source Ollama LLMs are good models in their own right, but the prompts used by LangChain's LLMGraphTransformer don't get the best out of these models.

IMO, LangChain's LLMGraphTransformer uses a very verbose prompt to explain to the LLM what documents to extract. You can take a look at how they construct the system prompts here. If you see the system prompt, it's got a lot of instructions, causing the smaller, lighter weight Ollama models to get lost in the instructions and failing to adhere to the prompts on many occasions. And because Kuzu requires that all relationships must only be between nodes that actually exist, there is a high chance of failure when not using OpenAI models.

As to why OpenAI models work most of the time, my hunch is that the OpenAI models are post-trained (during instruction-tuning) to adhere to very long and verbose prompts, so they don't get "lost in the middle" as much as the open source models do.

Suggestion

If I may, I can suggest alternate methods for you to much more reliably extract nodes/relationships from unstructured text, using BAML, and avoiding the use of LangChain and LLMGraphTransformer. If you have any representative data that you want me to try on your behalf, and the schema you're trying to model, I'd be happy to give you some starter code and show that it works with smaller, lighter-weight open source LLMs in Ollama too.

You could try and wrestle with LangChain's prompts in a way that you are able to improve the results, but IMO the level of abstraction that LangChain has isn't suited for general-purpose graph extraction with open source models. BAML is far better for this. Happy to help more, so please reach out to us via the discussion forum on Discord: https://kuzudb.com/chat and I can give you more concrete examples.

[wingraver]

Hi @prrao87, thanks a lot for the review, it really helps me to understand some of the challenges (and opportunities) in getting RAG right... especially graph RAG. I'll go over these takeaways again in a bit (haven't had my coffee yet) and see what additional insights I can glean to improve. Also, thanks for the offer, will definitely consider joining the Discord forum.