feat(tools): add surveys tools #117

ioannisj · 2025-09-04T14:50:54Z

Description

Towards #87

Adds a set of survey-related tools:

survey-create: Creates a new survey in the project.
survey-get: Get a specific survey by ID.
surveys-get-all: Get all surveys in the project with optional filtering.
survey-update: Update an existing survey by ID.
survey-delete: Delete a survey by ID (soft delete - marks as archived.

Plus:

surveys-global-stats: Get aggregated response statistics across all surveys in the project.
survey-stats: Get response statistics for a specific survey.

This is just CRUD operations mostly. Much more valuable response analysis tools will soon come from @marandaneto

Tried to follow tools documentation. Fair warning: this is my first time touching these tools and language, so excuse any naive bits of code

Tests?

Tested locally with lots of prompts and MCP Inspector tool
Added some integration tests

marandaneto · 2025-09-04T15:35:30Z

schema/tool-definitions.json

+		"category": "Surveys",
+		"summary": "Update an existing survey by ID."
+	},
+	"survey-delete": {


Do we call it 'delete' on the UI?
Or do we have both states? Arquive and really deleted?

In the UI we have "Delete" for all surveys (permanent) and "Archive" for stopped or completed surveys. Afaik, we don't have an actual soft delete.
Had the same concerns as this comment here, so I went with always archive from mcp instead

joshsny

This is looking great! Will give it a test run now locally - left some comments, main question is whether the LLM handles the input complexity well enough

joshsny · 2025-09-04T16:11:44Z

typescript/src/api/client.ts

+				const validatedInput = CreateSurveyInputSchema.parse(data);
+
+				const createResponseSchema = z.object({
+					id: z.string(),


Probably worth including more information about the created survey in the response here and in the other tools?

Done, returning the whole survey object which helps with "next steps" as well

joshsny · 2025-09-04T16:13:22Z

typescript/src/api/client.ts

+			delete: async ({
+				surveyId,
+				softDelete = true,
+			}: { surveyId: string; softDelete?: boolean }): Promise<


I think we can just make this a soft delete only to avoid things going too wrong

Added this optional param just for integration test cleanup (so we are left with a clean project).
The deleteHandler actually omits this param. My understanding is that the agent will always soft delete this way, right?

joshsny · 2025-09-04T16:16:01Z

typescript/src/schema/surveys.ts

+// Output schema - permissive for API responses
+export const SurveyQuestionOutputSchema = z.object({
+	type: z.string(),
+	question: z.string().optional(),


These should all use .nullish() to allow null and undefined values

Done. Thought optional() was doing it but reading up it doesn't actually include null values

joshsny · 2025-09-04T16:17:46Z

typescript/src/schema/surveys.ts

These look great and really detailed with some useful error messages - when you were testing this locally, was the LLM able to handle the complexity of the input? We might need to slightly restrict the kinds of surveys that can be made if not.

joshsny · 2025-09-04T16:18:31Z

typescript/src/schema/tool-inputs.ts

+});
+
+export const SurveyGetAllSchema = z.object({
+	data: ListSurveysInputSchema.optional(),


Can probably be just ListSurveysInputSchema directly

joshsny · 2025-09-04T16:19:30Z

typescript/src/tools/surveys/delete.ts

+	annotations: {
+		destructiveHint: true,
+		idempotentHint: false,
+		openWorldHint: false,


Suggested change

openWorldHint: false,

openWorldHint: true,

we should put this as true for this and the other tools

joshsny · 2025-09-04T16:20:40Z

typescript/src/tools/surveys/getAll.ts

+	}
+
+	// Format the surveys with better status display
+	const formattedSurveys = surveysResult.data.map((survey) => ({


can we pull this out into a function and then apply it elsewhere as well, so that we are being consistent with the output across tools?

Makes sense to include the url here too

Done, pulled into a utils function

joshsny · 2025-09-04T16:23:58Z

typescript/src/tools/surveys/update.ts

+	const { surveyId, ...data } = params;
+	const projectId = await context.stateManager.getProjectId();
+
+	if (data.questions) {


If we've got this logic here, it probably should also be in the create tool

Done. Single choice question branching is index based, but agent kept using the option text as the key. So my solution was to use Int in input schema, then map to string before making the call. I'm sure there's a better way though

joshsny

This is looking great! Will give it a test run now locally - left some comments, main question is whether the LLM handles the input complexity well enough

joshsny · 2025-09-04T17:07:04Z

Did some testing locally, a few thoughts:

The stats tools were not returning results for me, just the survey name, id and some other details, not sure if I was doing this wrong?
Surveys list works great
I got 'Permission Denied' whilst trying to use the create survey tool, I think it might just be struggling to get the input right as it's quite complex, I think we might need to simplify inputs here
Same for update, we probably need to simplify the input schema the same as the create one
Deleting surveys worked well

Let me know if you want any more testing :)

ioannisj

Hey @joshsny, thanks for reviewing this. I tried to address all the comments here.

Regarding the following

Did some testing locally, a few thoughts:

The stats tools were not returning results for me, just the survey name, id and some other details, not sure if I was doing this wrong?

This is based on responses, so for a fresh survey with no responses I guess it's because the response counts were zero? Tried running it for a survey with some responses and got some stats back

Surveys list works great

I got 'Permission Denied' whilst trying to use the create survey tool, I think it might just be struggling to get the input right as it's quite complex, I think we might need to simplify inputs here

Same for update, we probably need to simplify the input schema the same as the create one

When I was testing, Permission Denied was usually trying to create or update a survey to a project that was not in my Personal API key. I'm attaching some screenshots below with some prompts I tried. Looks like it's handing it okay but tbh I have no idea how to test this thoroughly.

This prompt here created the survey correctly with

Reasonable title and theme
Conditional logic on detractors with a reasonable follow up question
Correct display conditions

Then the follow-up prompt to link this to a feature flag seemed to work okay as well

Not sure if my prompts are just a bit too specific and giving it too much guidance though.

ioannisj · 2025-09-05T22:00:33Z

typescript/src/api/client.ts

+				const validatedInput = CreateSurveyInputSchema.parse(data);
+
+				const createResponseSchema = z.object({
+					id: z.string(),


Done, returning the whole survey object which helps with "next steps" as well

ioannisj · 2025-09-05T22:02:59Z

typescript/src/api/client.ts

+			delete: async ({
+				surveyId,
+				softDelete = true,
+			}: { surveyId: string; softDelete?: boolean }): Promise<


Added this optional param just for integration test cleanup (so we are left with a clean project).
The deleteHandler actually omits this param. My understanding is that the agent will always soft delete this way, right?

ioannisj · 2025-09-05T22:08:50Z

schema/tool-definitions.json

+		"category": "Surveys",
+		"summary": "Update an existing survey by ID."
+	},
+	"survey-delete": {


In the UI we have "Delete" for all surveys (permanent) and "Archive" for stopped or completed surveys. Afaik, we don't have an actual soft delete.
Had the same concerns as this comment here, so I went with always archive from mcp instead

ioannisj · 2025-09-05T22:16:22Z

typescript/src/schema/tool-inputs.ts

+});
+
+export const SurveyGetAllSchema = z.object({
+	data: ListSurveysInputSchema.optional(),


ioannisj · 2025-09-05T22:16:39Z

typescript/src/tools/surveys/delete.ts

+	annotations: {
+		destructiveHint: true,
+		idempotentHint: false,
+		openWorldHint: false,


ioannisj · 2025-09-05T22:40:46Z

typescript/src/tools/surveys/getAll.ts

+	}
+
+	// Format the surveys with better status display
+	const formattedSurveys = surveysResult.data.map((survey) => ({


Done, pulled into a utils function

ioannisj · 2025-09-05T22:45:40Z

typescript/src/tools/surveys/update.ts

+	const { surveyId, ...data } = params;
+	const projectId = await context.stateManager.getProjectId();
+
+	if (data.questions) {


Done. Single choice question branching is index based, but agent kept using the option text as the key. So my solution was to use Int in input schema, then map to string before making the call. I'm sure there's a better way though

ioannisj · 2025-09-05T23:12:54Z

typescript/src/schema/surveys.ts

+// Output schema - permissive for API responses
+export const SurveyQuestionOutputSchema = z.object({
+	type: z.string(),
+	question: z.string().optional(),


Done. Thought optional() was doing it but reading up it doesn't actually include null values

joshsny · 2025-09-06T14:46:34Z

Awesome - thanks again for this @ioannisj - looks great!

Since you are on paternity from Monday, I can merge this in next week - have a great time!

ioannisj added 19 commits August 27, 2025 19:38

feat: add crud schema

e916b6a

feat: add input schema

e09e297

feat: add handlers

53fa9bd

feat: add api client

4adf60f

feat: add surveys schema

c64b50b

feat: support branching logic

05a3364

fix: response based branching schemas

215d93e

feat: add more schema annotations

7b4b9e6

feat: add survey global stats tool

efaebcb

fix: survey create tool

ab43f7b

fix: improve survey-update tool

76f7b42

fix: add linked flag and variants to survey conditions

dcb5ddc

feat: add targeting filter

1bf31db

Merge branch 'main' into feat/tools/surveys

1c15a0c

fix: iteration count and frequency

6abcd1a

refactor: types

3e67b0d

fix: add some input validations

96eeb83

feat: add unit tests

58331ea

fix: tool definitions

7ab9698

ioannisj requested review from joshsny, JonathanLab and a team September 4, 2025 14:50

ioannisj marked this pull request as draft September 4, 2025 14:53

ioannisj added 2 commits September 4, 2025 17:57

chore: update schemas

b9dad6b

fix: typecheck

82c8b06

marandaneto reviewed Sep 4, 2025

View reviewed changes

joshsny reviewed Sep 4, 2025

View reviewed changes

fix: return full survey on create and update

6c2c18d

ioannisj added 5 commits September 6, 2025 01:14

fix: SurveyGetAllSchema

8945f74

fix: use openWorldHint true

2899b85

fix: format survey output

781b70f

fix: add create tool logic

78bd34e

fix: nullish for output schemas

819851d

ioannisj commented Sep 5, 2025

View reviewed changes

ioannisj requested a review from joshsny September 5, 2025 23:32

feat(tools): add surveys tools #117

Are you sure you want to change the base?

feat(tools): add surveys tools #117

Uh oh!

Conversation

ioannisj commented Sep 4, 2025

Description

Tests?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

joshsny left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

joshsny left a comment

Choose a reason for hiding this comment

Uh oh!

joshsny commented Sep 4, 2025

Uh oh!

ioannisj left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

joshsny commented Sep 6, 2025

Uh oh!

Uh oh!