[WIP] flowey: add new vmm-tests-run command by chris-oo · Pull Request #3167 · microsoft/openvmm

chris-oo · 2026-03-31T21:22:12Z

Add a new vmm-tests-run that accepts a nextest filter string, and only downloads the artifacts & resources needed for the tests identified before running the specified vmm test. This itself is a meta command that runs two other flowey commands, the existing vmm-tests which is used as a lower level command and a vmm-tests-discover command to create the list of artifacts needed by petri. This is due to the fact that we cannot dynamically create a pipeline from within a pipeline itself, so we need some kind of meta command in order to use the output from the artifact discovery for the actual vmm-tests command.

For now, there is a large table mapping petri artifacts to strings emitted by the petri binary. Enforce that this mapping is up to date by adding new CI passes to validate all cases are covered. This isn't ideal, but we can iterate further to at least unblock getting this basic command in.

Update the docs to note how to use this command, and note that vmm-tests is now a lower level command not normally intended to be used.

Copilot

Pull request overview

This PR adds an end-to-end local workflow for running vmm_tests via Flowey by introducing artifact discovery (from Petri test metadata), mapping discovered artifacts to build/download selections, and a convenience vmm-tests-run command that stitches discovery + run together.

Changes:

Add petri --list-required-artifacts JSON output (optionally scoped to specific tests via stdin) and Flowey jobs/pipelines to discover artifacts from a nextest filter.
Introduce artifact_to_build_mapping and CI validation to ensure all discovered Petri artifact IDs map to Flowey build/download selections.
Add cargo xflowey vmm-tests-run documentation and CLI plumbing for --artifacts-file-driven dependency minimization.

Reviewed changes

Copilot reviewed 22 out of 23 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
petri/src/test.rs	Emit machine-readable JSON of required/optional artifacts; add `--tests-from-stdin` for per-test querying.
Guide/src/dev_guide/tests/vmm.md	Update VMM test-running docs to prefer `vmm-tests-run` and describe targeting/cross-compilation.
Guide/src/dev_guide/dev_tools/xflowey.md	Document `vmm-tests-run` as a notable `cargo xflowey` pipeline.
flowey/flowey_lib_hvlite/src/lib.rs	Export the new artifact mapping module.
flowey/flowey_lib_hvlite/src/install_vmm_tests_deps.rs	Add `AutoInstall` request and allow skipping admin-only checks when not auto-installing.
flowey/flowey_lib_hvlite/src/download_openvmm_vmm_tests_artifacts.rs	Improve local download prompt behavior (terminal detection + timeout + explicit cancel).
flowey/flowey_lib_hvlite/src/artifact_to_build_mapping.rs	New mapping from Petri artifact ID strings to Flowey build selections + downloads (with unit tests).
flowey/flowey_lib_hvlite/src/_jobs/test_artifact_mapping_completeness.rs	New CI job to discover artifacts and fail if any are unmapped.
flowey/flowey_lib_hvlite/src/_jobs/mod.rs	Register new jobs (`local_discover_vmm_tests_artifacts`, `test_artifact_mapping_completeness`).
flowey/flowey_lib_hvlite/src/_jobs/local_discover_vmm_tests_artifacts.rs	New job that resolves nextest filters to test names and queries Petri for artifact JSON.
flowey/flowey_lib_hvlite/src/_jobs/local_build_and_run_nextest_vmm_tests.rs	Wire in `needs_release_igvm` and conditional release-IGVM download; add `BuildSelections::none()`.
flowey/flowey_lib_hvlite/src/_jobs/cfg_common.rs	Ensure `install_vmm_tests_deps` receives `AutoInstall` config.
flowey/flowey_lib_hvlite/Cargo.toml	Add `crossterm` dependency (events + Windows support).
flowey/flowey_hvlite/src/pipelines/vmm_tests.rs	Add `--artifacts-file` mode and resolve build/download selections from discovered artifact JSON.
flowey/flowey_hvlite/src/pipelines/vmm_tests_run.rs	New meta-command that runs discover then tests in sequence.
flowey/flowey_hvlite/src/pipelines/vmm_tests_discover.rs	New local-only pipeline to discover artifacts into a JSON file/stdout.
flowey/flowey_hvlite/src/pipelines/mod.rs	Register new subcommands; implement `VmmTestsRun` by executing directly (process exit).
flowey/flowey_hvlite/src/pipelines/checkin_gates.rs	Add GitHub CI job to validate artifact mapping completeness.
Cargo.lock	Lockfile updates for `crossterm` (and transitive deps).
.github/workflows/openvmm-pr.yaml	Regenerated workflow adding “test artifact mapping completeness” job.
.github/workflows/openvmm-pr-release.yaml	Regenerated workflow adding “test artifact mapping completeness” job in release CI.
.github/workflows/openvmm-ci.yaml	Regenerated workflow updates (bootstrap artifact uid bump + new mapping completeness job).
.github/copilot-instructions.md	Document `cargo xflowey vmm-tests-run` as the preferred dev validation path.

flowey/flowey_hvlite/src/pipelines/vmm_tests.rs

flowey/flowey_hvlite/src/pipelines/mod.rs

Guide/src/dev_guide/tests/vmm.md

flowey/flowey_lib_hvlite/src/download_openvmm_vmm_tests_artifacts.rs

petri/src/test.rs

github-actions · 2026-03-31T22:50:10Z

2 Petri tests failed (0 unstable)

Copilot

Pull request overview

Copilot reviewed 22 out of 23 changed files in this pull request and generated 4 comments.

Comments suppressed due to low confidence (1)

flowey/flowey_lib_hvlite/src/install_vmm_tests_deps.rs:121

This uses assert!(features.status.success()) on the result of an external command (DISM.exe). If DISM fails due to environment/state, this will panic instead of producing a recoverable error, which is undesirable for a CLI/pipeline tool. Replace the assert with anyhow::ensure!/bail! and include stderr/stdout in the error for diagnosis.

                        // Check if features are already enabled (requires admin, so skip if not auto_install)
                        if installing && auto_install && !features_to_enable.is_empty() {
                            let features = flowey::shell_cmd!(rt, "DISM.exe /Online /Get-Features").output()?;
                            assert!(features.status.success());
                            let features = String::from_utf8_lossy(&features.stdout).to_string();
                            let mut feature = None;

flowey/flowey_hvlite/src/pipelines/vmm_tests.rs

flowey/flowey_hvlite/src/pipelines/vmm_tests_run.rs

flowey/flowey_lib_hvlite/src/download_openvmm_vmm_tests_artifacts.rs

flowey/flowey_lib_hvlite/src/artifact_to_build_mapping.rs

Copilot

Pull request overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated 5 comments.

Comments suppressed due to low confidence (1)

flowey/flowey_lib_hvlite/src/install_vmm_tests_deps.rs:132

The DISM parsing path uses assert!(features.status.success()) / assert_eq! on command output. These can panic at runtime if DISM fails or output format changes. Prefer anyhow::ensure!/graceful error handling so the tool fails with a readable error instead of a panic.

                        // Check if features are already enabled (requires admin, so skip if not auto_install)
                        if installing && auto_install && !features_to_enable.is_empty() {
                            let features = flowey::shell_cmd!(rt, "DISM.exe /Online /Get-Features").output()?;
                            assert!(features.status.success());
                            let features = String::from_utf8_lossy(&features.stdout).to_string();
                            let mut feature = None;
                            for line in features.lines() {
                                if let Some((k, v)) = line.split_once(":") {
                                    if let Some(f) = feature {
                                        assert_eq!(k.trim(), "State");
                                        match v.trim() {
                                            "Enabled" => {
                                                assert!(features_to_enable.remove(f));
                                            }
                                            "Disabled" => {}
                                            _ => anyhow::bail!("Unknown feature enablement state"),
                                        }

petri/src/test.rs

flowey/flowey_lib_hvlite/src/artifact_to_build_mapping.rs

flowey/flowey_hvlite/src/pipelines/vmm_tests.rs

flowey/flowey_hvlite/src/pipelines/vmm_tests_discover.rs

Copilot

Pull request overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated 4 comments.

flowey/flowey_hvlite/src/pipelines/vmm_tests.rs

flowey/flowey_lib_hvlite/src/_jobs/local_discover_vmm_tests_artifacts.rs

Copilot · 2026-04-01T17:22:33Z

flowey/flowey_lib_hvlite/src/artifact_to_build_mapping.rs

+        // Validate target if present in the JSON
+        if let Some(ref file_target) = parsed.target {
+            let expected_target = format!(
+                "{}-{}",
+                match target_arch {
+                    target_lexicon::Architecture::X86_64 => "x86_64",
+                    target_lexicon::Architecture::Aarch64(_) => "aarch64",
+                    _ => "unknown",
+                },
+                match target_os {
+                    target_lexicon::OperatingSystem::Windows => "pc-windows-msvc",
+                    target_lexicon::OperatingSystem::Linux => "unknown-linux-gnu",
+                    _ => "unknown",
+                }
+            );
+
+            // Check if the target in the file is compatible with what we're building for
+            if !file_target.contains(expected_target.split('-').next().unwrap_or(""))
+                || (target_os == target_lexicon::OperatingSystem::Windows
+                    && !file_target.contains("windows"))
+                || (target_os == target_lexicon::OperatingSystem::Linux
+                    && !file_target.contains("linux"))
+            {
+                anyhow::bail!(
+                    "Target mismatch: artifacts file was generated for '{}', but building for '{}'",
+                    file_target,
+                    expected_target
+                );
+            }
+        }


The new target validation behavior is untested. Since this is parsing user-provided JSON files and bailing on mismatches, please add unit tests that (1) pass a JSON with a mismatched target and assert it errors, and (2) pass a matching target and assert it succeeds. This will help prevent regressions if the target normalization logic changes.

Copilot

Pull request overview

Copilot reviewed 21 out of 22 changed files in this pull request and generated 4 comments.

petri/src/test.rs

Copilot · 2026-04-01T17:38:03Z

flowey/flowey_lib_hvlite/src/install_vmm_tests_deps.rs

+                            FlowBackend::Local => auto_install.ok_or_else(|| {
+                                anyhow::anyhow!("Missing essential request: AutoInstall")
+                            })?,


AutoInstall is required for FlowBackend::Local unconditionally here, even when installing is false (e.g., the caller only requests GetCommands). Since the admin-only checks are already guarded by if installing && auto_install, you can avoid failing local GetCommands runs by only requiring AutoInstall when installing is true, or by defaulting auto_install to false when it’s not provided and no privileged actions are being performed.

Suggested change

FlowBackend::Local => auto_install.ok_or_else(|| {

anyhow::anyhow!("Missing essential request: AutoInstall")

})?,

FlowBackend::Local => {

if installing {

auto_install.ok_or_else(|| {

anyhow::anyhow!("Missing essential request: AutoInstall")

})?

} else {

// When not installing (e.g., only GetCommands), default to

// false if AutoInstall was not explicitly provided.

auto_install.unwrap_or(false)

}

}

flowey/flowey_hvlite/src/pipelines/vmm_tests.rs

Copilot · 2026-04-01T17:38:04Z

flowey/flowey_hvlite/src/pipelines/mod.rs

+            OpenvmmPipelines::VmmTestsRun(cmd) => {
+                // VmmTestsRun is a meta-command that runs discover + tests sequentially
+                // It doesn't return a pipeline, it executes directly
+                let result = cmd.run();
+                match result {
+                    Ok(()) => std::process::exit(0),
+                    Err(e) => {
+                        log::error!("{:?}", e);
+                        std::process::exit(1);
+                    }
+                }


Implementing VmmTestsRun by executing subprocesses and calling std::process::exit inside IntoPipeline breaks the expectation that this path constructs a pipeline. This can cause surprising behavior when generating pipelines/YAML for non-local backends (the command would run immediately instead of producing a pipeline), and it ignores backend_hint. Prefer making VmmTestsRun implement IntoPipeline (even if local-only) and bail! for non-local backends, or keep it out of the IntoPipeline/pipeline-subcommand surface entirely and expose it as a separate local-only entry point.

Suggested change

OpenvmmPipelines::VmmTestsRun(cmd) => {

// VmmTestsRun is a meta-command that runs discover + tests sequentially

// It doesn't return a pipeline, it executes directly

let result = cmd.run();

match result {

Ok(()) => std::process::exit(0),

Err(e) => {

log::error!("{:?}", e);

std::process::exit(1);

}

}

OpenvmmPipelines::VmmTestsRun(_cmd) => {

// VmmTestsRun is a meta-command that runs discover + tests sequentially.

// It is not suitable for constructing a declarative pipeline; use the

// dedicated local-only entry point instead of the pipeline interface.

anyhow::bail!(

"VmmTestsRun cannot be used to construct a pipeline; \

invoke the local vmm-tests-run command directly instead"

);

…uild those

…ified tests

Copilot

Pull request overview

Copilot reviewed 21 out of 22 changed files in this pull request and generated 2 comments.

Copilot · 2026-04-01T23:02:57Z

flowey/flowey_lib_hvlite/src/_jobs/local_discover_vmm_tests_artifacts.rs

+    let mut binary_path = None;
+
+    // Navigate to rust-suites -> vmm_tests::tests -> testcases
+    if let Some(vmm_tests) = json
+        .get("rust-suites")
+        .and_then(|s| s.get("vmm_tests::tests"))
+    {
+        // Get the binary path
+        if let Some(path) = vmm_tests.get("binary-path").and_then(|v| v.as_str()) {
+            binary_path = Some(PathBuf::from(path));
+        }
+
+        if let Some(testcases_obj) = vmm_tests.get("testcases").and_then(|t| t.as_object()) {
+            for (test_name, test_info) in testcases_obj {
+                // Check if filter-match.status == "matches"
+                let matches = test_info
+                    .get("filter-match")
+                    .and_then(|fm| fm.get("status"))
+                    .and_then(|s| s.as_str())
+                    == Some("matches");
+
+                if matches {
+                    test_names.push(test_name.clone());
+                }
+            }
+        }
+    }
+
+    let binary_path = binary_path
+        .ok_or_else(|| anyhow::anyhow!("Could not find test binary path in nextest output"))?;


parse_nextest_output is hard-coded to the rust-suites.vmm_tests::tests entry. The vmm_tests package contains multiple test binaries/suites (e.g. vmm_tests::tmks under vmm_tests/vmm_tests/tests/tmks.rs), so filters that match tests in other suites will be silently ignored and artifact discovery will be incomplete. Consider iterating over all entries in rust-suites (or otherwise discovering the relevant suite(s) dynamically), and if multiple binaries match, run --list-required-artifacts per binary and merge the results.

Suggested change

let mut binary_path = None;

// Navigate to rust-suites -> vmm_tests::tests -> testcases

if let Some(vmm_tests) = json

.get("rust-suites")

.and_then(|s| s.get("vmm_tests::tests"))

{

// Get the binary path

if let Some(path) = vmm_tests.get("binary-path").and_then(|v| v.as_str()) {

binary_path = Some(PathBuf::from(path));

}

if let Some(testcases_obj) = vmm_tests.get("testcases").and_then(|t| t.as_object()) {

for (test_name, test_info) in testcases_obj {

// Check if filter-match.status == "matches"

let matches = test_info

.get("filter-match")

.and_then(|fm| fm.get("status"))

.and_then(|s| s.as_str())

== Some("matches");

if matches {

test_names.push(test_name.clone());

}

}

}

}

let binary_path = binary_path

.ok_or_else(|| anyhow::anyhow!("Could not find test binary path in nextest output"))?;

let mut binary_path: Option<PathBuf> = None;

// Navigate to rust-suites and iterate all vmm_tests::* suites.

if let Some(rust_suites) = json.get("rust-suites").and_then(|s| s.as_object()) {

for (suite_name, suite_val) in rust_suites {

// We only care about suites from the vmm_tests package.

if !suite_name.starts_with("vmm_tests::") {

continue;

}

// Get the binary path for this suite.

if let Some(path_str) = suite_val.get("binary-path").and_then(|v| v.as_str()) {

let path = PathBuf::from(path_str);

match &binary_path {

None => {

binary_path = Some(path);

}

Some(existing) => {

if existing != &path {

return Err(anyhow::anyhow!(

"multiple vmm_tests binaries matched in nextest output: {:?} and {:?}",

existing,

path

));

}

}

}

}

if let Some(testcases_obj) = suite_val.get("testcases").and_then(|t| t.as_object()) {

for (test_name, test_info) in testcases_obj {

// Check if filter-match.status == "matches"

let matches = test_info

.get("filter-match")

.and_then(|fm| fm.get("status"))

.and_then(|s| s.as_str())

== Some("matches");

if matches {

test_names.push(test_name.clone());

}

}

}

}

}

let binary_path = binary_path.ok_or_else(|| {

anyhow::anyhow!("Could not find vmm_tests test binary path in nextest output")

})?;

Copilot · 2026-04-01T23:02:57Z

Guide/src/dev_guide/dev_tools/xflowey.md


 - `cargo xflowey build-igvm` - primarily dev-tool used to build OpenHCL IGVM files locally
 - `cargo xflowey restore-packages` - restores external packages needed to compile and run OpenVMM / OpenHCL
+- `cargo xflowey vmm-tests-run` - build and run VMM tests with automatic artifact discovery. Use `--filter "test(name)"` to run specific tests


This bullet uses escaped quotes inside inline code (--filter \"test(name)\"), which will render the backslashes literally in Markdown. It also exceeds the Guide style guide’s 80-column wrap. Suggest updating to --filter "test(name)" (without backslashes in the source) and wrapping the sentence onto a continuation line.

Suggested change

- `cargo xflowey vmm-tests-run` - build and run VMM tests with automatic artifact discovery. Use `--filter "test(name)"` to run specific tests

- `cargo xflowey vmm-tests-run` - build and run VMM tests with automatic

artifact discovery. Use `--filter "test(name)"` to run specific tests

smalis-msft · 2026-04-02T15:25:41Z

Could we not expose the vmm-tests command, and replace it with just running this new command with 'all'?

smalis-msft · 2026-04-02T15:28:46Z

we cannot dynamically create a pipeline from within a pipeline itself

we definitely should be able to do this, what issues were you hitting?

chris-oo requested a review from a team as a code owner March 31, 2026 21:22

Copilot AI review requested due to automatic review settings March 31, 2026 21:22

chris-oo requested a review from a team as a code owner March 31, 2026 21:22

github-actions bot added the Guide label Mar 31, 2026

chris-oo changed the title ~~Vmm tests run~~ [WIP] flowey: add new vmm-tests-run command Mar 31, 2026

Copilot started reviewing on behalf of chris-oo March 31, 2026 21:23 View session

Copilot AI reviewed Mar 31, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings March 31, 2026 22:49

Copilot started reviewing on behalf of chris-oo March 31, 2026 22:50 View session

Copilot AI reviewed Mar 31, 2026

View reviewed changes

chris-oo force-pushed the vmm-tests-run branch from cc34b8a to e76cdc7 Compare March 31, 2026 22:54

Copilot AI review requested due to automatic review settings March 31, 2026 23:29

Copilot started reviewing on behalf of chris-oo March 31, 2026 23:30 View session

Copilot AI reviewed Mar 31, 2026

View reviewed changes

chris-oo force-pushed the vmm-tests-run branch from bb87eb7 to 414d3a8 Compare March 31, 2026 23:44

Copilot AI review requested due to automatic review settings April 1, 2026 17:14

Copilot AI reviewed Apr 1, 2026

View reviewed changes

chris-oo force-pushed the vmm-tests-run branch from 04563d2 to 75b94c4 Compare April 1, 2026 17:25

Copilot started reviewing on behalf of chris-oo April 1, 2026 17:28 View session

Copilot AI review requested due to automatic review settings April 1, 2026 17:30

Copilot AI reviewed Apr 1, 2026

View reviewed changes

Copilot started reviewing on behalf of chris-oo April 1, 2026 17:49 View session

chris-oo added 7 commits April 1, 2026 15:50

flowey: add ability to query artifacts for tests being run and only b…

fb5e4e4

…uild those

flowey: meta command, remove mimic filter

e6c633d

flowey: remove nested somes

31f8bbd

flowey: add vmm-tests-run command to only download artifacts for spec…

57e14a0

…ified tests

flowey: add --skip-vhd-prompt command

0ec2dcf

flowey: add --skip-vhd-prompt to vmm-tests-run

2f4f791

flowey: add build-essential dep to artifact discovery

15ef74f

chris-oo added 5 commits April 1, 2026 15:54

flowey: check artifacts pass on each target

062c5fc

flowey: regen yaml

1709935

fixup: fixup rebase after dep removal

36ed380

flowey: add tpm artifact

f3018e6

feedback round 1

fb37378

Copilot AI review requested due to automatic review settings April 1, 2026 22:55

chris-oo force-pushed the vmm-tests-run branch from 85936aa to fb37378 Compare April 1, 2026 22:55

Copilot started reviewing on behalf of chris-oo April 1, 2026 22:57 View session

chris-oo changed the title ~~[WIP] flowey: add new vmm-tests-run command~~ flowey: add new vmm-tests-run command Apr 1, 2026

Copilot AI reviewed Apr 1, 2026

View reviewed changes

chris-oo changed the title ~~flowey: add new vmm-tests-run command~~ [WIP] flowey: add new vmm-tests-run command Apr 2, 2026

-                            FlowBackend::Local => auto_install.ok_or_else(|| {
-                                anyhow::anyhow!("Missing essential request: AutoInstall")
-                            })?,
+                            FlowBackend::Local => {
+                                if installing {
+                                    auto_install.ok_or_else(|| {
+                                        anyhow::anyhow!("Missing essential request: AutoInstall")
+                                    })?
+                                } else {
+                                    // When not installing (e.g., only GetCommands), default to
+                                    // false if AutoInstall was not explicitly provided.
+                                    auto_install.unwrap_or(false)
+                                }
+                            }

	- `cargo xflowey vmm-tests-run` - build and run VMM tests with automatic artifact discovery. Use `--filter "test(name)"` to run specific tests
	- `cargo xflowey vmm-tests-run` - build and run VMM tests with automatic
	artifact discovery. Use `--filter "test(name)"` to run specific tests

Conversation

chris-oo commented Mar 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Mar 31, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI Apr 1, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI Apr 1, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI Apr 1, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Apr 1, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 1, 2026

Choose a reason for hiding this comment

Uh oh!

smalis-msft commented Apr 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

smalis-msft commented Apr 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

chris-oo commented Mar 31, 2026 •

edited

Loading

smalis-msft commented Apr 2, 2026 •

edited

Loading

smalis-msft commented Apr 2, 2026 •

edited

Loading