RAG with LangchainJS: Session Resources

[multispark]

This is the resources and discussion thread for this live stream: https://aka.ms/raghack/js
It will be updated with links after the session.

Recording: https://www.youtube.com/watch?v=JCKXmABE7_U
Slides: https://github.com/microsoft/RAG_Hack/blob/main/slides/RAGHack-LangchainJS-Serverless.pptx
Blog: https://techcommunity.microsoft.com/t5/microsoft-developer-community/hack-together-rag-hack-building-rag-applications-with-langchain/ba-p/4238200

Please ask any follow-up questions here, and the speakers will see them.

[bhanukacodes]

Hi Team, thanks for the session.! I'm working on a chatbot project that implements the RAG pattern, and I've been referring to both the javascript (serverless-chat-langchainjs, azure-search-openai-javascript) and python (openai-demo) RAG blueprint projects. However, I have a few doubts and would greatly appreciate your feedback, specifically from the perspective of ensuring the best practices and recommendations for a production-grade application.

• For data ingestion, can we leverage integrated-vectorization by calling an AI Search API to index documents without using a custom implementation for handling document splitting, chunking, and indexing etc?
• Why is an orchestrator like LangChain framework is recommended compared to a simpler RAG implementation with Bot Framework?
• How can LangChain.js be used to preserve the state and context of a conversation to facilitate a context-aware conversational chat behavior?

[pamelafox]

I'll just reply to #1 since I have Python experience with the integrated vectorization feature, and let Yohan/Glaucia reply to the LangchainJS questions.
We do optionally provide a path in the azure-search-openai-demo repo for developers to use integrated vectorization. The benefits of that vectorization is that it's all cloud-based, can connect to many Azure stores, and can run on a schedule.

The drawbacks:

• The built-in splitting skill is based on characters, not token count. That's fine for English documents but not great if your documents are in CJK languages with much higher token:word ratios.
• The built-in indexing skill does not remember the PDF page number when it stores the page. That leads to a less optimal experience for the user since the citation can't jump to the relevant part of the page.
• Generally, it may be harder to customize if you run into other limitations.

[bhanukacodes]

Thanks for the detailed answer @pamelafox.

Regarding your second point under drawbacks, I would assume that we could potentially add a custom skill to address that? However, as you mentioned, a custom ingestion implementation offers more flexibility.

[sinedied]

An orchestrator like LangChain is mainly useful for its ability to abstract away the difference between different AI models, Vector DBs and such. That means you can easily switch from one provider to another to make experiments or reduce costs, like we're doing in this session to switch between local models and DB and Azure models and DB. It also provides many components to load data from different sources if needed (YT videos, Office docs, GH repos, etc).

For preserving context, Langchain.js has this concept of "memory" that you can use to store/restore chat history and use it to answer questions. You can see an example here: https://js.langchain.com/v0.2/docs/how_to/qa_chat_history_how_to/#chain-with-chat-history

It's not implemented yet in our JS sample, but it's in our backlog.

[pamelafox]

@bhanukacodes Yep, you should be able to bring in a custom skill instead, if you're comfortable building skills. I believe they're implemented as Azure Functions.

[bhanukacodes]

Thanks you very much @sinedied and @pamelafox for the valuable information shared.!

[pamelafox]

Here's a great blog post from Glaucia summarizing the session:
https://techcommunity.microsoft.com/t5/microsoft-developer-community/hack-together-rag-hack-building-rag-applications-with-langchain/ba-p/4238200

[DMK1961]

In Github Codespaces, After entering 'azd up' I am getting the following error:

ERROR: error executing step command 'provision': deployment failed: error deploying infrastructure: deploying to subscription:

Deployment Error Details:
InvalidTemplateDeployment: The template deployment 'openai' is not valid according to the validation procedure. The tracking id is 'a7e493c3-b851-4a25-b827-4df3708a6b77'. See inner errors for details.
InsufficientQuota: This operation require 15 new capacity in quota Tokens Per Minute (thousands) - GPT-4-Turbo, which is bigger than the current available capacity 8. The current quota usage is 0 and the quota limit is 8 for quota Tokens Per Minute (thousands) - GPT-4-Turbo.

TraceID: e450e5023599b0d113ad04506ee663e9

I have submitted a quota increase request. Not sure whether the same will be processed in time.

Thanks.

[sinedied]

In the meantime, you can try reducing the capacity to 8 here: https://github.com/Azure-Samples/serverless-chat-langchainjs/blob/main/infra/main.bicep#L45 and redeploying, this should fix your issues.

Once your quota has been updated, you can safely change the capacity and redeploy again.

[DMK1961]

Thanks. Yes.

[DMK1961]

Yohan, Thanks for your prompt reply. Will check it out at the earliest. DMK1961 On Monday 16 September, 2024 at 05:37:20 pm IST, Yohan Lasorsa ***@***.***> wrote: In the meantime, you can try reducing the capacity to 8 here: https://github.com/Azure-Samples/serverless-chat-langchainjs/blob/main/infra/main.bicep#L45 and redeploying, this should fix your issues. Once your quota has been updated, you can safely change the capacity and redeploy again. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: ***@***.***>

[DMK1961]

Yohan,

Thanks a lot. That worked.

[negar-mir]

Hi @sinedied
I wanted to follow the tutorial for my learning. in creating-a-new-azure-functions-project suggested to start with starter branch but I can't see it in the list of branches.

could you please check how to access starter branch?

Best regards,
Negar

[sinedied]

Hi! The tutorial is currently work in progress and not up to date, that's why the starter branch does not exist yet.
@glaucia86 is currently working on it.

Meanwhile, it's not as detailed, but we have this blog post explaining how the sample works.

[negar-mir]

thanks