ci: daily Evals CI for Extensions/Skills on github using Evalbench#152
Open
omkargaikwad23 wants to merge 19 commits intomainfrom
Open
ci: daily Evals CI for Extensions/Skills on github using Evalbench#152omkargaikwad23 wants to merge 19 commits intomainfrom
omkargaikwad23 wants to merge 19 commits intomainfrom
Conversation
omkargaikwad23
added
the
do not merge
omkargaikwad23
force-pushed
the
ci-evals
branch
2 times, most recently
from
4004d05 to
e69c7b6
Compare
… testing of Cloud SQL PostgreSQL extension
…el configurations
…d variables with $PROJECT_ID
…cess, and remove evalbench patching scripts
…anager for improved security
omkargaikwad23
force-pushed
the
ci-evals
branch
from
f0abf12 to
73bb202
Compare
omkargaikwad23
changed the title
… to use daily CI evaluation database
…uation server execution script
… it was attempting to use gemini-3-flash-preview and failing due to a 404 permissions error.
…valuation dataset scenarios
…pdate Cloud Build symlinks,
omkargaikwad23
commented
Apr 21, 2026
| steps: | ||
|
|
||
| # --- Evaluation Step --- | ||
| - name: 'us-central1-docker.pkg.dev/cloud-db-nl2sql/evalbench/eval_server:89aa9fefd4b247610a95ef0896ba55d468563f50' |
Author
There was a problem hiding this comment.
Will use latest image once recently pushed changes in evalbench are deployed.
https://github.com/GoogleCloudPlatform/evalbench/pull/336/changes
To identify which evaluation results belong to my extension added extension_id: cloud-sql-postgresql key
omkargaikwad23
removed
the
do not merge
| scorers: | ||
| trajectory_matcher: {} | ||
| goal_completion: | ||
| model_config: /workspace/evals/gemini_2.5_pro_model.yaml |
Contributor
There was a problem hiding this comment.
Add all the required scorers
… scorers to run configuration
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.