Implement initial version of Hypervisor crate (abstraction between KVM and Microsoft Hyper-V) #1225

russell-islam · 2020-05-23T07:11:34Z

This is the initial implementation of the proposal that we made earlier. The purpose of this crate is to make the hypervisor support abstracted to include Microsoft Hyper-V. We will have a hypervisor trait that will be separately implemented by KVM and Hyper-V. Based on the hypervisor we will create the VMfs and VcpuFD which are also some genetic traits. These two traits will be implemented by KVM and HyperV. For KVM there will be two implementations KvmVMfd and KvmVcpuFd and same for HyperV.

Sample code: hypervisor = KvmHyperVisor::new()
vm = hypervisor.create_vm()
vcpu = vm.create_vcpu(id)

vm.get/set()
vcpu.get/set()

For the payload we have a binding inside the hypervisor crate that will abstract between HyperV and KVM.

The plan is to move most of the KVM reference to Hypervisor crate. The hypervisor will hand the payload based on KVM or HyperV. This is the initial proposal. For the payload I have followed the previous commit by crowd-strike.

russell-islam · 2020-05-23T18:33:30Z

Looks like build on aarch64 fails. Is there any way to build aarch64 binary on x86-64 machine?

michael2012z · 2020-05-25T02:27:01Z

@russell-islam , you can cross-build for aarch64 on x86_64, that's what we do with the 2 "Cross Build" checks now.

To try cross-build locally, you need to install target aarch64-unknown-linux-gnu or aarch64-unknown-linux-musl, and a tool-chain (to link).

michael2012z · 2020-05-25T02:45:28Z

About the build error on aarch64. It was caused by an issue in cloud-hypervisor/kvm-bindings on aarch64, see this PR for the complete story: cloud-hypervisor/kvm-bindings#1.

As a workaround, you can modify the Cargo.toml (which uses cloud-hypervisor/kvm-bindings with with-serde feature) when it comes to aarch64 build, here is an example: https://github.com/cloud-hypervisor/cloud-hypervisor/blob/master/.github/workflows/cross-build.yaml#L26

Changing it run: sed -i 's/"with-serde",\ //g' vmm/Cargo.toml && sed -i 's/"with-serde",\ //g' hypervisor/Cargo.toml should fix the error.

yisun-git · 2020-05-26T03:04:04Z

Hi,

We proposed the Hypervisor crate in cloud-hypervisor and rust-vmm before.
rust-vmm/community#50
https://github.intel.com/cloud-hypervisor/community/issues/2

The draft codes had been provided at:
https://github.com/yisun-git/Hypervisor_on_FC/tree/dbg_yisun1_hypervisor

But the community has different opinions on this. So, we stopped the work.

I want to learn what is the purpose of this PR? Have you addressed the comments in above issues?

Thanks,
Sun Yi

russell-islam · 2020-05-26T04:01:28Z

@russell-islam , you can cross-build for aarch64 on x86_64, that's what we do with the 2 "Cross Build" checks now.

To try cross-build locally, you need to install target aarch64-unknown-linux-gnu or aarch64-unknown-linux-musl, and a tool-chain (to link).

Is this documented somewhere with detailed steps?

russell-islam · 2020-05-26T04:03:21Z

Hi,

We proposed the Hypervisor crate in cloud-hypervisor and rust-vmm before.
rust-vmm/community#50
https://github.intel.com/cloud-hypervisor/community/issues/2

The draft codes had been provided at:
https://github.com/yisun-git/Hypervisor_on_FC/tree/dbg_yisun1_hypervisor

But the community has different opinions on this. So, we stopped the work.

I want to learn what is the purpose of this PR? Have you addressed the comments in above issues?

Thanks,
Sun Yi

@yisun-git Thanks for the information. The purpose of this PR is to get initial comments on the Hypervisor crate that will eventually support Microsoft Hyper-V.

russell-islam · 2020-05-27T01:24:08Z

Hi,

We proposed the Hypervisor crate in cloud-hypervisor and rust-vmm before.
rust-vmm/community#50
https://github.intel.com/cloud-hypervisor/community/issues/2

The draft codes had been provided at:
https://github.com/yisun-git/Hypervisor_on_FC/tree/dbg_yisun1_hypervisor

But the community has different opinions on this. So, we stopped the work.

I want to learn what is the purpose of this PR? Have you addressed the comments in above issues?

Thanks,
Sun Yi

@yisun-git Thanks for the information. The purpose of this PR is to get initial comments on the Hypervisor crate that will eventually support Microsoft Hyper-V. I would love to seek your contribution too.

russell-islam · 2020-05-27T01:50:26Z

Aarch64 build is fine now but I am getting the following now. Looks like this a known issue even with latest master:
error[E0277]: the trait bound kvm_bindings::x86::bindings_v4_20_0::kvm_sregs: serde::Serialize is not satisfied

vmm/Cargo.toml

hypervisor/Cargo.toml

michael2012z · 2020-05-27T03:47:34Z

@russell-islam , see michael2012z#28, attention to the last commit.

russell-islam · 2020-05-27T03:59:38Z

@russell-islam , see michael2012z#28, attention to the last commit.

Thank you. I misunderstood the fix. Now I got it.

sameo

Thanks @russell-islam for that PR. Even though the overall design looks good, it needs a bit more work and I added some comments accordingly.

On a more general note, we need to rework the commits structure as well. The current commit history shows how you iteratively improve the code into something that builds and works. What we need is a commit history that shows the structure of the work and how each piece of the puzzle gets added together. For example:

Commit #1: Add the Vcpu trait
Commit #2: Add the Hypervisor trait
Commit #3: Add the KVM bindings and implementation for both traits
Commit #4: Port the current code to use both traits and build under the kvm feature.

This will make it clearer and easier to review as well. Finally, for each of those commits:

The commit subject should be formatted according to what's documented in https://github.com/cloud-hypervisor/cloud-hypervisor/blob/master/CONTRIBUTING.md
Those are fundamental changes, so we expect the commit message to describe the rationale behind the code changes and how it's implemented.

arch/src/x86_64/interrupts.rs

hypervisor/src/lib.rs

hypervisor/src/bindings.rs

vfio/src/vfio_pci.rs

vmm/src/cpu.rs

hypervisor/src/lib.rs

hypervisor/src/cpuidpatch.rs

russell-islam · 2020-05-29T16:42:29Z

Thanks a lot @sameo . I will address your comments.

sboeuf

This is a first pass on this PR, I'll dig into more details tomorrow or next week.
A few nits about your commit messages:

Try to keep the commit messages title short (refer to only one component before the : helps)
Some commit messages body have some indentation, make sure you remove it
Don't hesitate to be verbose in the commit message body :)

sboeuf · 2020-06-04T18:14:55Z

Cargo.toml

 clap = { version = "2.33.1", features=["wrap_help"] }
 epoll = ">=4.0.1"
 futures = { version = "0.3.5", features = ["thread-pool"] }
+hypervisor = { path = "hypervisor" }


Could you please add an extra commit introducing the new hypervisor crate without anything inside?
This commit should be the first of the entire patchset, and it will be the opportunity to describe the goal of this new crate in the commit message.

This commit should be the first of the entire patchset, and it will be the opportunity to describe the goal of this new crate in the commit message.

@russell-islam initially introduced the hypervisor trait ad the first commit, but since it depends on the cpu and vm traits, it had a few placeholders that added some noise when actually introducing the cpu and vmm traits further down the PR. I suggested to introduce traits in the dependency order, i.e. CPU -> VM -> Hypervisor.

@sameo I guess I can add one more extra commit just to add the skeleton of the hypervisor crate. vcpu would be the next commit.

hypervisor/src/kvm.rs

sboeuf · 2020-06-04T18:25:37Z

hypervisor/src/kvm.rs

+    ///             exit_reason => panic!("unexpected exit reason: {:?}", exit_reason),
+    ///         }
+    ///     }
+    /// }


I'm wondering if all these examples/documentation are needed since they've been ported from the kvm-ioctls crate. That feels like a lot of duplicates.

Right. But the example has some changes reflecting the current hypervisor implementations.

Right, but could we keep it shorter since the core content is already part of kvm-ioctls?

Will do . thanks

russell-islam · 2020-06-05T03:44:40Z

I will make the change once I get some reviews to avoid back and forth update.

sameo · 2020-06-05T06:06:45Z

Cargo.toml

 clap = { version = "2.33.1", features=["wrap_help"] }
 epoll = ">=4.0.1"
 futures = { version = "0.3.5", features = ["thread-pool"] }
+hypervisor = { path = "hypervisor" }


This commit should be the first of the entire patchset, and it will be the opportunity to describe the goal of this new crate in the commit message.

@russell-islam initially introduced the hypervisor trait ad the first commit, but since it depends on the cpu and vm traits, it had a few placeholders that added some noise when actually introducing the cpu and vmm traits further down the PR. I suggested to introduce traits in the dependency order, i.e. CPU -> VM -> Hypervisor.

sameo · 2020-06-05T06:14:54Z

hypervisor/src/cpu.rs

+// Copyright © 2019 Intel Corporation
+//
+// SPDX-License-Identifier: Apache-2.0
+//


There should not be 2 SPDX ids. Because the crowdstrike code was dual licensed, we can choose to keep this crate dual licensed or make it Apache 2.0 only. The former would allow for reusing that code with less permissive licenses, so that would be my preference.

@russell-islam Are you ok with dual licensing apache or MIT or do you want to only keep the apache license?

Guru is in a talk within Msft. I will update you once I hear something.

sameo · 2020-06-05T06:19:21Z

hypervisor/src/x86_64/mod.rs

+pub use {
+    kvm_bindings::kvm_lapic_state as LapicState,
+    kvm_bindings::kvm_xcrs as ExtendedControlRegisters, kvm_bindings::kvm_xsave as Xsave,
+    kvm_bindings::CpuId, kvm_bindings::Msrs as MsrEntries, kvm_ioctls::Cap, kvm_ioctls::Kvm,


Just a nit, but I find it more readable to have one line per export:

pub use { kvm_bindings::kvm_lapic_state as LapicState, kvm_bindings::kvm_xcrs as ExtendedControlRegisters, kvm_bindings::kvm_xsave as Xsave, kvm_bindings::CpuId, kvm_bindings::Msrs as MsrEntries, kvm_ioctls::Cap, kvm_ioctls::Kvm, }

I need to remove some VisCode code fmt. This is auto saved. :(

This is not accepted by cargo fmt check. I had to revert it. :(

sameo · 2020-06-05T06:21:18Z

hypervisor/src/x86_64/mod.rs

+pub use {
+    kvm_bindings::kvm_lapic_state as LapicState,
+    kvm_bindings::kvm_xcrs as ExtendedControlRegisters, kvm_bindings::kvm_xsave as Xsave,
+    kvm_bindings::CpuId, kvm_bindings::Msrs as MsrEntries, kvm_ioctls::Cap, kvm_ioctls::Kvm,


I think Cap and Kvm are not x86 specific. You may want to export them from the top level lib.rs file.

sameo · 2020-06-05T06:28:11Z

hypervisor/src/vm.rs

+    /// Registers an event to be signaled whenever a certain address is written to.
+    fn register_ioevent(
+        &self,
+        fd: &EventFd,


Just a note here, and I'm not requesting any change: This is Linux specific and the Hyper-V implementation will have to change at least this method to something more generic.
My point is: This PR is a first step and we need to acknowledge that this traits may change as we add support for other hypervisors later on.

Yes. I totally agree. We have to adjust the trait once we have hyperv-ioctls ready.

sameo · 2020-06-05T09:09:47Z

arch/src/x86_64/mod.rs

 use crate::RegionType;
-use kvm_bindings::CpuId;
-use kvm_ioctls::*;
+


Unneeded change.

sameo · 2020-06-05T09:13:46Z

arch/src/x86_64/interrupts.rs

 use byteorder::{LittleEndian, ReadBytesExt, WriteBytesExt};

-use kvm_bindings::kvm_lapic_state;
+use hypervisor::kvm_lapic_state;


This should be use hypervisor::x86_64::LapicState and then replace all kvm_lapic_state instances with LapicState

sameo · 2020-06-05T09:14:51Z

arch/src/x86_64/mod.rs

 use crate::InitramfsConfig;
 use crate::RegionType;
-use kvm_bindings::CpuId;
-use kvm_ioctls::*;


use hypervisor::CpuId would remove the need for some of the below changes.

sameo · 2020-06-05T09:32:34Z

arch/src/x86_64/gdt.rs


 // For GDT details see arch/x86/include/asm/segment.h
-use kvm_bindings::kvm_segment;
+use hypervisor::kvm_segment;


Same here, you want to use hypervisor::x86_64::SegmentRegister

sameo · 2020-06-05T09:35:14Z

hypervisor/src/lib.rs

+#[cfg(target_arch = "x86_64")]
+pub use kvm_bindings::{kvm_lapic_state, kvm_segment, kvm_userspace_memory_region};
+pub use kvm_ioctls;
+pub use kvm_ioctls::VcpuExit;


This should be exported through the kvm module only.

likebreath · 2020-06-18T20:44:23Z

@russell-islam I played your PR with GDB and took a reference from the working master branch. It looks like the VMM (cloud-hypervisor) is not hanging: vmm/http-servier/vcpu/virtio threads are working fine. What is hanging seems to be the guest which is stuck at the early stage of booting. I also see the following error from the clh debug log, which are MMIO access errors. It aligns with our hypothesis that the problem is tied to MMIO. I will take a closer look tomorrow. Anything you found interesting, please just share/ask.

  1cloud-hypervisor: 6.049099ms: INFO:vmm/src/memory_manager.rs:732 -- Created userspace mapping: 0 -> 7fffa8000000 40000000                                                                                       
  2cloud-hypervisor: 683.317154ms: WARN:vm-virtio/src/transport/mmio.rs:382 -- unknown virtio mmio register write: 0x50                                                                                            
  3cloud-hypervisor: 683.379447ms: WARN:vm-virtio/src/transport/mmio.rs:382 -- unknown virtio mmio register write: 0x50                                                                                            
  4cloud-hypervisor: 683.409303ms: WARN:vm-virtio/src/transport/mmio.rs:382 -- unknown virtio mmio register write: 0x50                                                                                            
  5cloud-hypervisor: 683.438271ms: WARN:vm-virtio/src/transport/mmio.rs:382 -- unknown virtio mmio register write: 0x50                                                                                            
  6cloud-hypervisor: 683.467305ms: WARN:vm-virtio/src/transport/mmio.rs:382 -- unknown virtio mmio register write: 0x50                                                                                            
  7cloud-hypervisor: 683.496437ms: WARN:vm-virtio/src/transport/mmio.rs:382 -- unknown virtio mmio register write: 0x50                                                                                            
  8cloud-hypervisor: 683.525402ms: WARN:vm-virtio/src/transport/mmio.rs:382 -- unknown virtio mmio register write: 0x50                                                                                            
  9cloud-hypervisor: 683.554339ms: WARN:vm-virtio/src/transport/mmio.rs:382 -- unknown virtio mmio register write: 0x50                                                                                            
 10cloud-hypervisor: 683.589732ms: WARN:vm-virtio/src/transport/mmio.rs:382 -- unknown virtio mmio register write: 0x50                                                                                            
 11cloud-hypervisor: 683.618777ms: WARN:vm-virtio/src/transport/mmio.rs:382 -- unknown virtio mmio register write: 0x50
...

russell-islam · 2020-06-18T21:37:06Z

unknown virtio mmio register write

Thanks @likebreath. Looks like the offset actually notifying the queue index

QueueNotify 0x050 W	Queue notifier Writing a queue index to this register notifies the device that there are new buffers to process in the queue.

sameo · 2020-06-18T23:57:25Z

@russell-islam You were registering the wrong io events and not getting guest notifications. That:
sameo@c07ff47

should fix it.

likebreath · 2020-06-19T03:10:32Z

@russell-islam I tried the patch from @sameo, and now clh with MMIO feature boots correctly. :)

I think Sam's branch is here https://github.com/sameo/cloud-hypervisor-1/commits/topic/hypervisor and the updated patch I tried is here: sameo@cad17ff.

sameo · 2020-06-19T07:44:47Z

@russell-islam I tried the patch from @sameo, and now clh with MMIO feature boots correctly. :)

I think Sam's branch is here https://github.com/sameo/cloud-hypervisor-1/commits/topic/hypervisor and the updated patch I tried is here: sameo@cad17ff.

@likebreath is correct, sameo@cad17ff is an implementation that's correct, although there may be more idiomatic way to do it.

The purpose of this trait is to add support for other hypervisors than KVM, like e.g. Microsoft Hyper-V. Further commits will define additional hypervisor related traits like Vcpu and Vm. Each of the supported hypervisor will need to implement all traits defined from the hypervisor crate. Signed-off-by: Muminul Islam <muislam@microsoft.com>

likebreath · 2020-06-19T20:10:14Z

Great. The CI now is passing. Now only build failing with aarch64 w/ musl.

This Vcpu trait should be implemented by each underlying hypervisor. Previously created hypervisor object should create the VM based on already selected hypervisor and Vm object should create this vcpu object based on same hyperviosr. Each of this object should be referenced by trait object i.e <dyn Vcpu>. Signed-off-by: Muminul Islam <muislam@microsoft.com> Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>

This VM trait should be implemented by each underlying hypervisor. Previously created hypervisor object should create the VM based on already selected hypervisor. This is just the trait definition. For each of supported hypervisor we need to implement the trait. Later we will implement this trait for KVM and then Microsoft Hyper-V. Signed-off-by: Muminul Islam <muislam@microsoft.com> Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>

As the only hypervisor that Cloud-Hypervisor supports is KVM, the Hypervisor trait accomodates for the upcoming KVM implementation. This trait will be instanciated at build time through hypervisor specific features, i.e. it's not aiming at run-time selection of hypervisors for Cloud-Hypervisor. Signed-off-by: Muminul Islam <muislam@microsoft.com> Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>

For each of the traits we are defining kvm related structures and add the trait implementation to the structs. For more information please see the kvm-ioctls and kvm-bindings crate. This is a standalone implementation that does not include the switch of the Cloud-Hypervisor vmm and arch crates to it. Signed-off-by: Muminul Islam <muislam@microsoft.com> Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>

Implement the vCPU state getter and setter separately from the initial KVM Hypervisor trait implementation, mostly for readability purposes. Signed-off-by: Muminul Islam <muislam@microsoft.com> Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>

Start moving the vmm, arch and pci crates to being hypervisor agnostic by using the hypervisor trait and abstractions. This is not a complete switch and there are still some remaining KVM dependencies. Signed-off-by: Muminul Islam <muislam@microsoft.com> Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>

russell-islam · 2020-06-19T22:34:13Z

Great. The CI now is passing. Now only build failing with aarch64 w/ musl.

@likebreath thanks for keeping an aye. The integration tests passed. yeah!!

sameo · 2020-06-20T09:30:16Z

@sboeuf Are you happy with the changes? Please have a look. As a first step towards a complete hypervisor absraction, this PR is ready to be reviewed imho.

sboeuf

The PR looks good!
I have two remaining comments (can be addressed in a follow up PR if needed).
Both the list of expected caps and the list of MSRs are specific to Cloud-Hypervisor, which is why I think they should live in the vmm crate instead of the hypervisor one (which is supposed to be pushed to rust-vmm eventually).

sboeuf · 2020-06-22T06:22:33Z

hypervisor/src/kvm/x86_64/mod.rs

+pub fn check_required_kvm_extensions(kvm: &Kvm) -> KvmResult<()> {
+    if !kvm.check_extension(Cap::SignalMsi) {
+        return Err(KvmError::CapabilityMissing(Cap::SignalMsi));
+    }
+    if !kvm.check_extension(Cap::TscDeadlineTimer) {
+        return Err(KvmError::CapabilityMissing(Cap::TscDeadlineTimer));
+    }
+    if !kvm.check_extension(Cap::SplitIrqchip) {
+        return Err(KvmError::CapabilityMissing(Cap::SplitIrqchip));
+    }
+    Ok(())


This is very Cloud-Hypervisor specific. I mean the requirement for having these KVM caps supported is not something that all VMM might need, so it'd be more appropriate to let the VMM check them.

In this case we need to have a way to return this kvm object reference back to vmm.

sboeuf · 2020-06-22T06:27:03Z

hypervisor/src/kvm/x86_64/mod.rs

+    MsrEntries::from_entries(&[
+        msr!(msr_index::MSR_IA32_SYSENTER_CS),
+        msr!(msr_index::MSR_IA32_SYSENTER_ESP),
+        msr!(msr_index::MSR_IA32_SYSENTER_EIP),
+        msr!(msr_index::MSR_STAR),
+        msr!(msr_index::MSR_CSTAR),
+        msr!(msr_index::MSR_LSTAR),
+        msr!(msr_index::MSR_KERNEL_GS_BASE),
+        msr!(msr_index::MSR_SYSCALL_MASK),
+        msr!(msr_index::MSR_IA32_TSC),
+        msr_data!(
+            msr_index::MSR_IA32_MISC_ENABLE,
+            msr_index::MSR_IA32_MISC_ENABLE_FAST_STRING as u64
+        ),
+        msr_data!(msr_index::MSR_MTRRdefType, MTRR_ENABLE | MTRR_MEM_TYPE_WB),


A bit the same idea as the caps here, but I'd like to mention this list of MSRs is specific to Cloud-Hypervisor.

cloud-hypervisor#1225 introduces a hypervisor abstraction crate, which breaks some of the unit test cases on AArch64. This commit fixes related test cases. Signed-off-by: Henry Wang <henry.wang@arm.com>

#1225 introduces a hypervisor abstraction crate, which breaks some of the unit test cases on AArch64. This commit fixes related test cases. Signed-off-by: Henry Wang <henry.wang@arm.com>

bonzini · 2020-09-22T17:22:18Z

Out of curiosity, will the Hypervisor implementation for Hyper-V support Linux as well, or will it be only for Windows (similar to the WHPX support in QEMU)?

sameo · 2020-09-22T17:54:17Z

Out of curiosity, will the Hypervisor implementation for Hyper-V support Linux as well, or will it be only for Windows (similar to the WHPX support in QEMU)?

It will support KVM and Hyper-V on Linux.

vsrivatsa1 · 2021-06-08T15:27:41Z

Hello, what's the status of this work being merged into rust-vmm?

michael2012z mentioned this pull request May 25, 2020

Fix the build error with "with-serde" feature in CH branch (on AArch64) cloud-hypervisor/kvm-bindings#1

Merged

russell-islam force-pushed the muislam/untangle-part1 branch from 0893ea7 to eeab12b Compare May 27, 2020 01:23

russell-islam force-pushed the muislam/untangle-part1 branch from eeab12b to e3129de Compare May 27, 2020 01:25

michael2012z reviewed May 27, 2020

View reviewed changes

vmm/Cargo.toml Outdated Show resolved Hide resolved

michael2012z reviewed May 27, 2020

View reviewed changes

hypervisor/Cargo.toml Outdated Show resolved Hide resolved

russell-islam force-pushed the muislam/untangle-part1 branch from e3129de to f9be7c5 Compare May 27, 2020 03:32

russell-islam force-pushed the muislam/untangle-part1 branch from f9be7c5 to d0626cb Compare May 27, 2020 03:58

russell-islam changed the title ~~[RFC] Initial proposal of Hypervisor crate (abstraction between KVM and HyperV)~~ [RFC] Initial proposal of Hypervisor crate (abstraction between KVM and Microsoft HyperV) May 27, 2020

russell-islam changed the title ~~[RFC] Initial proposal of Hypervisor crate (abstraction between KVM and Microsoft HyperV)~~ [RFC] Initial proposal of Hypervisor crate (abstraction between KVM and Microsoft Hyper-V) May 27, 2020

sameo reviewed May 29, 2020

View reviewed changes

russell-islam force-pushed the muislam/untangle-part1 branch 2 times, most recently from a3c1f56 to d7229b5 Compare June 4, 2020 00:39

sboeuf requested changes Jun 4, 2020

View reviewed changes

sameo reviewed Jun 5, 2020

View reviewed changes

russell-islam force-pushed the muislam/untangle-part1 branch from d7229b5 to 361fa67 Compare June 8, 2020 04:31

russell-islam changed the title ~~[RFC] Initial proposal of Hypervisor crate (abstraction between KVM and Microsoft Hyper-V)~~ Implement initial version of Hypervisor crate (abstraction between KVM and Microsoft Hyper-V) Jun 8, 2020

russell-islam force-pushed the muislam/untangle-part1 branch 2 times, most recently from fc82da7 to ab81277 Compare June 8, 2020 19:31

russell-islam force-pushed the muislam/untangle-part1 branch 2 times, most recently from 224300d to c5fe9da Compare June 18, 2020 20:18

russell-islam force-pushed the muislam/untangle-part1 branch 2 times, most recently from 4316afa to c5fe9da Compare June 18, 2020 23:20

russell-islam force-pushed the muislam/untangle-part1 branch from c5fe9da to 8823fcb Compare June 19, 2020 19:14

russell-islam added 6 commits June 19, 2020 13:36

russell-islam force-pushed the muislam/untangle-part1 branch from 8823fcb to a8b4a15 Compare June 19, 2020 21:41

sameo approved these changes Jun 20, 2020

View reviewed changes

sboeuf approved these changes Jun 22, 2020

View reviewed changes

sameo merged commit e4dee57 into cloud-hypervisor:master Jun 22, 2020

Implement initial version of Hypervisor crate (abstraction between KVM and Microsoft Hyper-V) #1225

Implement initial version of Hypervisor crate (abstraction between KVM and Microsoft Hyper-V) #1225

Uh oh!

Conversation

russell-islam commented May 23, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

russell-islam commented May 23, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

michael2012z commented May 25, 2020

Uh oh!

michael2012z commented May 25, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yisun-git commented May 26, 2020

Uh oh!

russell-islam commented May 26, 2020

Uh oh!

russell-islam commented May 26, 2020

Uh oh!

russell-islam commented May 27, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

russell-islam commented May 27, 2020

Uh oh!

Uh oh!

Uh oh!

michael2012z commented May 27, 2020

Uh oh!

russell-islam commented May 27, 2020

Uh oh!

sameo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

russell-islam commented May 29, 2020

Uh oh!

sboeuf left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

russell-islam commented Jun 5, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

russell-islam Jun 5, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

russell-islam commented May 23, 2020 •

edited

Loading

russell-islam commented May 23, 2020 •

edited

Loading

michael2012z commented May 25, 2020 •

edited

Loading

russell-islam commented May 27, 2020 •

edited

Loading

russell-islam Jun 5, 2020 •

edited

Loading

russell-islam Jun 5, 2020 •

edited

Loading