Sequential API Function Calling Using GraphQL Schema
Abstract: Function calling using Large Language Models (LLMs) is an active research area that aims to empower LLMs with the ability to execute APIs to perform real-world tasks. However, sequential function calling using LLMs with interdependence between functions is still under-explored. To this end, we introduce GraphQLRestBench, a dataset consisting of natural language utterances paired with function call sequences representing real-world REST API calls with variable mapping between functions. In order to represent the response structure of the functions in the LLM prompt, we use the GraphQL schema of the REST APIs. We also introduce a custom evaluation framework for our dataset consisting of four specially designed metrics. We evaluate three open-source code LLMs on our dataset using few-shot Chain-of-Thought and ReAct prompting to establish a reasonable baseline.
Paper Type: Short
Research Area: Resources and Evaluation
Research Area Keywords: benchmarking, NLP datasets, evaluation methodologies, evaluation
Contribution Types: NLP engineering experiment, Data resources
Languages Studied: English
Submission Number: 2295
Loading