Dynamic Handling of ConfirmationRequest and ConfirmationResponse

[codebuster007]

I tried using the ConditionalAwaitingTool to wait for a users confirmation before a tool is executed. However it does not provide a mechanism to capture the users response via an llm. One would need to handle that manually. Usually outside of an agent process to convert the users response to an instance of ConfirmationResponse. This defeats the purpose of leveraging ai to handle the response accordingly. Here is my suggestion below. Would love to hear your thoughts on this approach:

package com.embabel.experiments.tools;

import com.embabel.agent.api.tool.DelegatingTool;
import com.embabel.agent.api.tool.Tool;
import com.embabel.agent.api.tool.ToolCallContext;
import com.embabel.agent.core.AgentProcess;
import com.embabel.agent.core.Blackboard;
import com.embabel.agent.core.hitl.AwaitableResponseException;
import com.embabel.agent.core.hitl.ConfirmationRequest;
import com.embabel.agent.core.hitl.ConfirmationResponse;
import com.fasterxml.jackson.annotation.JsonPropertyDescription;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.jspecify.annotations.NonNull;

import java.time.Instant;
import java.util.UUID;

public class ConfirmationAwaitingTool implements DelegatingTool {
    private final Tool delegate;
    private final String message;
    private final ObjectMapper objectMapper;
    private final Tool.Definition confirmDefinition;

    public ConfirmationAwaitingTool(Tool delegate, String message, ObjectMapper objectMapper) {
        this.delegate = delegate;
        this.message = message;
        this.objectMapper = objectMapper;
        this.confirmDefinition = Tool.Definition.create(
                delegate.getDefinition().getName(),
                "Confirm or cancel: " + delegate.getDefinition().getDescription(),
                Tool.InputSchema.of(ConfirmInput.class)
        );
    }

    @Override
    public @NonNull Tool getDelegate() {
        return delegate;
    }

    @Override
    public Tool.@NonNull Definition getDefinition() {
        return isPending() ? confirmDefinition : delegate.getDefinition();
    }

    @Override
    public Tool.@NonNull Metadata getMetadata() {
        return delegate.getMetadata();
    }

    @Override
    public Tool.@NonNull Result call(@NonNull String input, @NonNull ToolCallContext context) {
        if (isPending()) {
            return handleConfirmation(input, context);
        }
        return handleAwaiting(input, context);
    }

    private Tool.Result handleAwaiting(String input, ToolCallContext context) {
        var agentProcess = AgentProcess.get();
        if (agentProcess == null) throw new IllegalStateException("No AgentProcess available");

        var blackboard = agentProcess.getBlackboard();
        var lastRequest = blackboard.last(ConfirmationRequest.class);
        var lastResponse = blackboard.last(ConfirmationResponse.class);

        boolean alreadyConfirmed = lastRequest != null
                && lastResponse != null
                && lastResponse.getAwaitableId().equals(lastRequest.getId())
                && lastResponse.getAccepted();

        if (!alreadyConfirmed) {
            throw new AwaitableResponseException(
                    new ConfirmationRequest<>(input, message)
            );
        }
        return delegate.call(input, context);
    }

    private Tool.Result handleConfirmation(String input, ToolCallContext context) {
        var agentProcess = AgentProcess.get();
        var blackboard = agentProcess.getBlackboard();
        ConfirmationRequest pending = blackboard.last(ConfirmationRequest.class);
        try {
            ConfirmInput confirmInput = objectMapper.readValue(input, ConfirmInput.class);
            agentProcess.plusAssign(new ConfirmationResponse(
                    UUID.randomUUID().toString(),
                    pending.getId(),
                    confirmInput.accepted(),
                    false,
                    Instant.now()
            ));

            if (confirmInput.accepted()) {
                return delegate.call((String) pending.getPayload(), context);
            }
            return Tool.Result.text("Action cancelled");

        } catch (JsonProcessingException e) {
            throw new RuntimeException(e);
        }
    }

    private boolean isPending() {
        var agentProcess = AgentProcess.get();
        if (agentProcess == null) return false;
        var pending = agentProcess.getBlackboard().last(ConfirmationRequest.class);
        return pending != null
                && !isResolved(agentProcess.getBlackboard(), pending);
    }

    private boolean isResolved(Blackboard blackboard, ConfirmationRequest<?> request) {
        var response = blackboard.last(ConfirmationResponse.class);
        return response != null && response.getAwaitableId().equals(request.getId());
    }

    public record ConfirmInput(
            @JsonPropertyDescription("whether to confirm or not")
            Boolean accepted){}
}

This can then be used like below:

Tool createTask = Tool.fromFunction(
                "createTask",
                "Create a task.",
                TaskRequest.class,
                TaskResponse.class,
                input -> {
                    log.info("Creating task: {}", input.title());
                    return "Task with title %s created".formatted(input.title());
                }
 );


Tool confirmTaskCreation = new ConfirmationAwaitingTool(createTaskRequest,
        "Do you want me to proceed with this task creation?", objectMapper);

[johnsonr]

This looks worthwhile. Would love to see a PR and discuss there.

[codebuster007]

This looks worthwhile. Would love to see a PR and discuss there.

Thanks. To cater for instances where the user needs to make updates to the initial input, I made some updates to this. I'm using the existing TypedTool as I find its typing allows creating the correct schema for the input. Renamed it also from ConfirmationAwaitingTool to ConfirmationAwareTool. See below java impl. I can create a pr for this in kotlin.

package com.embabel.experiments.tools;

import com.embabel.agent.api.tool.Tool;
import com.embabel.agent.api.tool.ToolCallContext;
import com.embabel.agent.api.tool.TypedTool;
import com.embabel.agent.core.AgentProcess;
import com.embabel.agent.core.Blackboard;
import com.embabel.agent.core.hitl.AwaitContext;
import com.embabel.agent.core.hitl.Awaitable;
import com.embabel.agent.core.hitl.AwaitableResponseException;
import com.embabel.agent.core.hitl.ConfirmationRequest;
import com.embabel.agent.core.hitl.ConfirmationResponse;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.node.ObjectNode;

import java.time.Instant;
import java.util.UUID;

public class ConfirmationAwareTool<I, O> extends TypedTool<I, O> {

    private final Tool.Definition confirmDefinition;
    private final java.util.function.Function<AwaitContext, Awaitable<?, ?>> confirmationDecider;

    public ConfirmationAwareTool(
            String name, String description,
            Class<I> inputType, Class<O> outputType,
            ObjectMapper objectMapper,
            java.util.function.Function<I, O> executor,
            java.util.function.Function<AwaitContext, Awaitable<?, ?>> confirmationDecider
    ) {
        super(name, description, inputType, outputType, Tool.Metadata.DEFAULT, objectMapper, executor);
        this.confirmationDecider = confirmationDecider;

        // Merge accepted into the normal schema — no separate ConfirmInput type needed
        this.confirmDefinition = Tool.Definition.create(
                name,
                description + "\nA confirmation is pending. Set accepted=true to proceed " +
                        "(with the same or updated parameters), or accepted=false to cancel.",
                super.getDefinition().getInputSchema().withParameter(
                        new Tool.Parameter("accepted", Tool.ParameterType.BOOLEAN,
                                "Confirm (true) or cancel (false)", true, null, null, null)
                )
        );
    }

    @Override
    public Tool.Definition getDefinition() {
        return isPending() ? confirmDefinition : super.getDefinition();
    }

    @Override
    public Tool.Result call(String input, ToolCallContext context) {
        if (isPending()) {
            return handleConfirmation(input);
        }
        var agentProcess = AgentProcess.get();
        if (agentProcess != null) {
            var awaitable = confirmationDecider.apply(new AwaitContext(input, agentProcess, this));
            if (awaitable != null) throw new AwaitableResponseException(awaitable);
        }
        return super.call(input); // TypedTool: parse → typedCall → convertResult
    }

    private Tool.Result handleConfirmation(String input) {
        var agentProcess = AgentProcess.get();
        var blackboard = agentProcess.getBlackboard();
        @SuppressWarnings("unchecked")
        ConfirmationRequest<I> pending = (ConfirmationRequest<I>) blackboard.last(ConfirmationRequest.class);

        try {
            ObjectNode node = (ObjectNode) getObjectMapper().readTree(input);
            boolean accepted = node.path("accepted").asBoolean(true);
            node.remove("accepted");

            agentProcess.plusAssign(new ConfirmationResponse(
                    UUID.randomUUID().toString(), pending.getId(), accepted, false, Instant.now()
            ));

            if (accepted) {
                // Use re-submitted fields (may include user's changes); fall back to stored if empty
                I payload = node.isEmpty()
                        ? pending.getPayload()
                        : getObjectMapper().treeToValue(node, getInputType());
                return convertResult(typedCall(payload));
            }
            return Tool.Result.text("Action cancelled.");
        } catch (JsonProcessingException e) {
            throw new RuntimeException(e);
        }
    }

    private boolean isPending() {
        var agentProcess = AgentProcess.get();
        if (agentProcess == null) return false;
        var pending = agentProcess.getBlackboard().last(ConfirmationRequest.class);
        return pending != null && !isResolved(agentProcess.getBlackboard(), pending);
    }

    private boolean isResolved(Blackboard blackboard, ConfirmationRequest<?> request) {
        var response = blackboard.last(ConfirmationResponse.class);
        return response != null && response.getAwaitableId().equals(request.getId());
    }
}

This can then be used as so:

Tool createTask = new ConfirmationAwareTool<>(
                "createTask",
                "Create a task on the Task Board. Returns the created task with its ID.",
                TaskRequest.class,
                TaskResponse.class,
                objectMapper,
                request -> {
                    log.info("Created task request: {}", request);
                    return request.taskResponse();
                },                                        // executor
                awaitCtx -> new ConfirmationRequest<>(awaitCtx.getInput(), "Please review task details and confirm.") // decider
        );

---

Another thing which I noticed and its probably for another issue, the json schema generated on input parameters does not include the description set using @JsonPropertyDescription. I updated this on the TypedBaseInputSchema.kt and the descriptions were added, this further aided the correctness of the input generated by the llm:

for (field in type.declaredFields) {
                if (java.lang.reflect.Modifier.isStatic(field.modifiers)) continue
                params.add(
                    Tool.Parameter(
                        name = field.name,
                        type = mapPropertyTypeToParameterType(field.type),
                        description = field.getAnnotation(JsonPropertyDescription::class.java)?.value ?: field.name, // added getting description from annotation, fallsback to field.name
                        required = true,
                    )
                )
            }

[igordayen]

@codebuster007 - as suggested by @johnsonr - could you please try to raise PR, thank you