Jekyll2024-10-07T05:29:07+00:00https://sathnaga86.com/feed.xmlLinux, Cloud, DevOps and Beyond - A Deep Dive into Testing!Blog that would concentrate on software testing techniques and tools for Linux, KVM, Cloud areas and DevOps.Satheesh Rajendransathnaga86@gmail.comBooting Powerpc Guest From Virtio Fs2020-05-12T00:00:00+00:002020-05-12T00:00:00+00:00https://sathnaga86.com/2020/05/12/booting-powerpc-guest-from-virtio-fsHow to boot a PowerPC KVM guest from virtio-fs

Introduction:

Virtio-fs is a shared file system that lets virtual machines access a directory tree on the host. Unlike existing approaches, it is designed to offer local file system semantics and performance. Virtio-fs was started at Red Hat and is being developed in the Linux, QEMU, FUSE, and Kata Containers communities that are affected by code changes. More details on virtio-fs can be found here.

This blog gives a step by step guide on how to setup an PowerPC KVM Environment to boot a KVM guest from virtio-fs as rootfs.

Environment used:

Steps:

1. Create a working directory on host.

  • mkdir -p /home/virtio-fs
  • cd /home/virtio-fs

    For rest of the blog this folder is referred as $PWD

2. Needs a Guest kernel that supports virtio-fs as bootdisk, let’s build one.

Get kernel:

  • git clone https://github.com/sathnaga/linux -b virtio-fs

    Mainline kernel has support for virtio-fs already, my above branch has two additional patches to get rootfs feature and kernel configs to enable it for my defconfig used below:

    • Patch from https://patchwork.kernel.org/patch/11134865/mbox to get virtio-fs as rootfs support.

    • Patch to enable virtio fs kernel configs to support on guest kernel.

Compile kernel:

  • cd linux
  • make ppc64le_guest_defconfig
  • make -j 120 -s
  • cd ..

Now we have guest kernel at $PWD/linux/vmlinux

3. Needs a qemu that supports virtio-fs, let’s build one.

Get Qemu:

Compile Qemu:

  • cd qemu
  • ./configure --target-list=ppc64-softmmu --enable-debug --disable-werrormake
  • make -j 120 -s
  • cd ..

Now we have qemu binary at $PWD/qemu/ppc64-softmmu/qemu-system-ppc64

4. Needs a virtiofs daemon, let’s build one.

  • cd qemu
  • make -j 8 virtiofsd
  • cd ..

Now we have virtio fs daemon binary at $PWD/qemu/virtiofsd

5. Let’s build a Fedora root file system based on F32:

  • mkdir $PWD/virtio-fs-root

  • dnf --installroot=$PWD/virtio-fs-root --releasever=32 install system-release vim-minimal systemd passwd dnf rootfiles pciutils

    Select yes to install packages.

    Incase to install additional packages, use below commandline,

    dnf –installroot=$PWD/virtio-fs-root –releasever=32 install “packages to install”

  • openssl passwd -1 123456

    $1$sWUb0J.r$cyjCLYRfstQ0xEHVZ45UZ/

    let’s create a passwd for guest root file system, in above command 123456 is our guest password and output generated is md5 password of same, change the password as per your need and use the generated one in shadow file.

  • Edit password field of $PWD/virtio-fs-root/etc/shadow with above md5sum password.

    Before edit:

    $ head -1 $PWD/virtio-fs-root/etc/shadow

    root:*:18292:0:99999:7:::

    After edit:

    $ head -1 $PWD/virtio-fs-root/etc/shadow root:$1$sWUb0J.r$cyjCLYRfstQ0xEHVZ45UZ/:18292:0:99999:7:::

  • mknod $PWD/virtio-fs-root/dev/hvc0 c 1 2

    This creates serial device for guest console

  • cp $PWD/virtio-fs-root/usr/lib/systemd/system/serial-getty@.service $PWD/virtio-fs-root/etc/systemd/system/getty.target.wants/serial-getty@hvc0.service
  • cd $PWD/virtio-fs-root/etc/systemd/system/getty.target.wants/
  • ln -s serial-getty@hvc0.service getty@hvc0.service
  • Edit getty@hvc0.service with below changes 
      ...
  [Unit]
  ....
  ConditionPathExists=/dev/hvc0
  ....
  [Install]
  WantedBy=getty.target
  DefaultInstance=hvc0

    this is needed to get the guest serial console working fine.

  • cd /home/virtio-fs

6. Let’s Boot PowerPC KVM guest using qemu cmdline with virtio-fs.

  • $PWD/qemu/virtiofsd -o --socket-path=/tmp/vhostqemu -o source=$PWD/virtio-fs-root -o cache=none &

    this command starts the virtio fs daemon in background

  • Run below qemu command to start a KVM guest 
    $PWD/qemu/ppc64-softmmu/qemu-system-ppc64 \
      -m 4096 -object memory-backend-file,id=mem,size=4G,mem-path=/dev/shm,share=on -numa node,memdev=mem \
      -smp 4 -enable-kvm -serial mon:stdio -vga none -nographic \
      -kernel $PWD/linux/vmlinux \
      -append "rootfstype=virtiofs root=myfs rw" \
      -chardev socket,id=char0,path=/tmp/vhostqemu \
      -device vhost-user-fs-pci,queue-size=1024,chardev=char0,tag=myfs

Login using the password created in previous steps, this blog used 123456 as guest root password, let’s use that to login, below are command outputs inside kvm guest.

localhost login:
localhost login: root
Password:
Last login: Tue May 12 14:30:59 on hvc0
[   16.610692] id (134) used greatest stack depth: 10736 bytes left
[root@localhost ~]#
# lscpu
Architecture:                    ppc64le
Byte Order:                      Little Endian
CPU(s):                          4
On-line CPU(s) list:             0-3
Thread(s) per core:              1
Core(s) per socket:              1
Socket(s):                       4
NUMA node(s):                    1
Model:                           2.3 (pvr 004e 1203)
Model name:                      POWER9 (architected), altivec supported
Hypervisor vendor:               KVM
Virtualization type:             para
L1d cache:                       128 KiB
L1i cache:                       128 KiB
NUMA node0 CPU(s):               0-3
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Mitigation; RFI Flush, L1D private per thread
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Mitigation; RFI Flush, L1D private per thread
Vulnerability Spec store bypass: Mitigation; Kernel entry/exit barrier (eieio)
Vulnerability Spectre v1:        Mitigation; __user pointer sanitization, ori31
                                 speculation barrier enabled
Vulnerability Spectre v2:        Mitigation; Software count cache flush, Softwar
                                 e link stack flush
Vulnerability Tsx async abort:   Not affected

[root@localhost ~]# cat /etc/os-release
NAME=Fedora
VERSION="32 (Thirty Two)"
ID=fedora
VERSION_ID=32
VERSION_CODENAME=""
PLATFORM_ID="platform:f32"
PRETTY_NAME="Fedora 32 (Thirty Two)"
ANSI_COLOR="0;34"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:32"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f32/system-administrators-guide/"
SUPPORT_URL="https://fedoraproject.org/wiki/Communicating_and_getting_help"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=32
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=32
PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"
[root@localhost ~]# uname -a
Linux localhost 5.7.0-rc5-g7aa8a99a5-dirty #5 SMP Tue May 12 07:33:17 EDT 2020 ppc64le ppc64le ppc64le GNU/Linux
[root@localhost ~]# lspci
00:00.0 Mass storage controller: Red Hat, Inc. Device 105a (rev 01)

Now We have PowerPC KVM guest booted with virtio-fs as bootdisk.

This blog is written based on below reference documents on virtio-fs and my additional steps that are needed for setup on PowerPC environment.

Thanks for reading, Hope this blog helps you in someways :-)

]]>Satheesh Rajendransathnaga86@gmail.comGuide To Run Test On Power9 From Laptop Using Simulator2019-11-19T00:00:00+00:002019-11-19T00:00:00+00:00https://sathnaga86.com/2019/11/19/guide-to-run-test-on-power9-from-laptop-using-simulatorGuide to run commands/tests on a IBM Power9 using systemsim(mambo) from your Laptop using op-test and buildroot

Env Used:

  • Intel(R) Core(TM) i5-5300U CPU @ 2.30GHz
  • Fedora 31(5.3.7-301.fc31.x86_64)

Pre-requisite:

Install packages needed for cross compilation

For Ubuntu env:
    apt-get install gcc-powerpc64le-linux-gnu gcc valgrind \
    expect libssl-dev device-tree-compiler make \
    xz-utils
For Fedora env:
    dnf install gcc-powerpc64le-linux-gnu \
    binutils-powerpc64-linux-gnu gcc make diffutils findutils \
    expect valgrind-devel dtc openssl-devel xz

How to:

  1. Clone skiboot, Cross compile and get skiboot.lid.

  2. Clone linux kernel, Cross compile and get vmlinux. 
     ARCH=powerpc CROSS_COMPILE=powerpc64le-linux-gnu- make ppc64le_defconfig
 ARCH=powerpc CROSS_COMPILE=powerpc64le-linux-gnu- make -j `nproc`

  3. Clone buildroot, Compile and get rootfs image.

    • run ‘make menuconfig’.

    • Choose

      Target options —> Target Architecture (PowerPC64 (little endian)) Target Architecture Variant (power8)

      Filesystem images —> cpio the root filesystem (for use as an initial RAM filesystem)`

      and other packages you would need.

    • run make.
    • Find root filesystem in output/images(rootfs.cpio).

  4. Install systemsim(mambo) rpm for power9 and check for systemsim(mambo) binary for power9 under /opt/ibm/systemsim-p9/run/p9/.

  5. Create op-test config to run OpTestMamboBuildRoot test, same test can modified to run different commands inside power9 host systemsim. 
     $cat mambo_p9.cfg
 [op-test]
 bmc_type=mambo
 mambo_binary=/opt/ibm/systemsim-p9/run/p9/power9;5dc2cbf4
 flash_skiboot=/home/satheesh/github/skiboot/skiboot.lid
 flash_kernel=/home/satheesh/github/linux/vmlinux
 flash_initramfs=/home/satheesh/github/buildroot/output/rootfs.cpio
 host_user=root
 host_password=
  6. Run OpTestMamboBuildRoot test, which executes cat /proc/interrupts and cat /proc/cpuinfo inside a power9 host systemsim(mambo) env as shown below. 
    
 $./op-test -c mambo_p9.cfg --run testcases.OpTestMamboBuildRoot.OpTestMamboBuildRoot

 ...
 ...
 runTest (testcases.OpTestMamboBuildRoot.OpTestMamboBuildRoot) ... Using initial run script skiboot.tcl
 No network support selected^M
 INFO: 0: (0): !!!!!! Simulator now in TURBO mode !!!!!!
 [    0.000170223,5] OPAL v6.5-86-g1c282887 starting...^M
 [    0.000176398,7] initial console log level: memory 7, driver 5^M
 [    0.000181940,6] CPU: P9 generation processor (max 4 threads/core)^M
 [    0.000187332,7] CPU: Boot CPU PIR is 0x0000 PVR is 0x004e1200^M
 [    0.000194047,7] OPAL table: 0x30113630 .. 0x30113ba0, branch table: 0x30002000^M
 [    0.000202048,7] Assigning physical memory map table for nimbus^M
 [    0.000208137,7] FDT: Parsing fdt @0x1f00000^M
 [    0.001023665,5] Enabling Mambo console^M
 [    0.001027953,5] CHIP: Detected Mambo simulator^M
 [    0.001096322,5] CHIP: Chip ID 0000 type: P9N DD2.30^M
 [    0.001346958,5] PLAT: Detected Mambo platform^M
 [    0.001558571,5] CPU: All 1 processors called in...^M
 [    0.001600800,3] SBE: Master chip ID not found.^M
 [    0.001767201,3] NVRAM: Partition at offset 0x0 has incorrect 0 length^M
 [    0.001773195,3] NVRAM: Re-initializing (size: 0x00040000)^M
 [    0.001935098,5] STB: secure boot not supported^M
 [    0.001943298,5] STB: trusted boot not supported^M
 [    0.002078002,4] FLASH: Can't load resource id:0. No system flash found^M
 [    0.002095952,4] FLASH: Can't load resource id:1. No system flash found^M

 ...
 ...
 [console-expect]#lsprop /sys/firmware/devicetree/base/ibm,firmware-versions
 lsprop /sys/firmware/devicetree/base/ibm,firmware-versions
 -sh: lsprop: not found^M^M
 [console-expect]#
 date
 which whoami && whoami

 [console-expect]#date
 which whoami && whoami
 Tue Nov 19 07:03:56 UTC 2019^M^M
 [console-expect]#/usr/bin/whoami^M^M
 root^M^M
 [console-expect]#echo $?
 echo $?
 0^M^M
 [console-expect]#stty -echo
 stty -echo
 [console-expect]#echo $?
 echo $?
 0^M^M
 [console-expect]#uname -a
 uname -a
 Linux buildroot 5.3.0-rc7 #1 SMP Wed Sep 4 14:34:53 IST 2019 ppc64le GNU/Linux^M^M
 [console-expect]#echo $?
 echo $?
 0^M^M
 [console-expect]#cat /proc/cpuinfo
 cat /proc/cpuinfo
 processor	: 0^M^M
 cpu		: POWER9, altivec supported^M^M
 clock		: 512.000000MHz^M^M
 revision	: 2.0 (pvr 004e 1200)^M^M
 ^M^M
 timebase	: 512000000^M^M
 platform	: PowerNV^M^M
 model		: Mambo,Simulated-System^M^M
 machine		: PowerNV Mambo,Simulated-System^M^M
 firmware	: OPAL^M^M
 MMU		: Radix^M^M
 [console-expect]#echo $?
 echo $?
 0^M^M
 [console-expect]#cat /proc/interrupts
 cat /proc/interrupts
            CPU0       ^M^M
  16:          0      XICS   2 Edge      IPI^M^M
  17:          0  OPAL EVT  11 Level     opal-msg^M^M
  18:         21  OPAL EVT   4 Edge      hvc_console^M^M
 LOC:      33666   Local timer interrupts for timer event device^M^M
 BCT:          0   Broadcast timer interrupts for timer event device^M^M
 LOC:          0   Local timer interrupts for others^M^M
 SPU:          0   Spurious interrupts^M^M
 PMI:          0   Performance monitoring interrupts^M^M
 MCE:          0   Machine check exceptions^M^M
 HMI:          0   Hypervisor Maintenance Interrupts^M^M
 NMI:          0   System Reset interrupts^M^M
 WDG:          7   Watchdog soft-NMI interrupts^M^M
 DBL:          0   Doorbell interrupts^M^M
 [console-expect]#echo $?
 echo $?
 0^M^M
 ok

 ----------------------------------------------------------------------
 Ran 1 test in 28.541s

 OK

Thanks for Reading…, Hope this blog helps you in someways :-)

]]>Satheesh Rajendransathnaga86@gmail.comLong Haul And Multi Config Kvm Tests Using Avocado Vt2019-06-20T00:00:00+00:002019-06-20T00:00:00+00:00https://sathnaga86.com/2019/06/20/long-haul-and-multi-config-KVM-tests-using-avocado-VTLong Haul and multi configuration KVM guests testing using Avocado-VT

With reducing time to market, it becomes crucial we attempt the long haul and stress tests mimicking the real world scenarios, and it is difficult to do it manually with less time in hand. So, here comes the proven KVM test suite Avocado-VT, where we have implemented recently a lot of capabilities which will allow any one to run not only longevity tests but also in parallel multiple Guest VM with different configurations.

My previous blog explain how to configure and use Avocado-VT KVM test suites.

Here let us see how to run some of long haul and multi configuration tests:

Scenario 1: User wants to check whether guest withstand huge number of reboot requests, lets take some really huge number, 5000.

Solution: Here comes the simple avocado command line allow us to accomplish it,

avocado run io-github-autotest-qemu.reboot \
--vt-type libvirt \
--vt-guest-os JeOS.27.ppc64le \
--job-results-dir /home/sath \
--vt-extra-params qemu_binary=/home/sath/qemu/ppc64-softmmu/qemu-system-ppc64 \
emulator_path=//home/sath/qemu/ppc64-softmmu/qemu-system-ppc64 \
mem=81920 smp=256 vcpu_maxcpus=256 \
take_regular_screendumps=no backup_image_before_testing=no \
libvirt_controller=virtio-scsi scsi_hba=virtio-scsi-pci drive_format=scsi-hd \
use_os_variant=no restore_image_after_testing=no \
vga=none display=nographic \
kernel=/home/sath/linux/vmlinux kernel_args='root=/dev/sda2 rw console=tty0 console=ttyS0,115200 init=/sbin/init initcall_debug selinux=0' \
virtinstall_qemu_cmdline=' -M pseries,ic-mode=xive' \
initrd='' \
use_serial_login=yes \
kill_vm=yes kill_vm_libvirt=yes env_cleanup=yes create_vm_libvirt=yes \
test_timeout=1440000 \
reboot_count=5000

JOB ID     : 73013bc0a7102829a87786bb6ba3efd35e816cc0
JOB LOG    : /home/sath/job-2019-05-18T08.17-73013bc/job.log
 (1/1) io-github-autotest-qemu.reboot: PASS (201977.56 s)
RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB TIME   : 201979.43 s
JOB HTML   : /home/sath/job-2019-05-18T08.17-73013bc/results.html

Explain on this test w.r.t command line parameters:

This test reboots a KVM guest booted with given upstream kernel and qemu parameters
in libvirt environment for user defined iterations, in this case `reboot_count=5000`
times on given upstream qemu.

Above test ran for almost 56 hours, i.e more than 2 days, Avocado-VT framework is stable enough to facilitate such environment and tests.

Scenario 2: User wants to check similar events say reboot (it can be any vm events like cpu hotplug, vcpu pining, suspend/resume or all together) in parallel for multiple guest with different configurations, like different machine types, cpu, memory setting etc.

Solution: Another simple avocado command line allow us to accomplish it,

avocado run multivm_cpustress.custom_host_events.custom_vm_events \
--vt-type libvirt \
--vt-guest-os JeOS.27.ppc64le \
--job-results-dir /home/sath \
--vt-extra-params qemu_binary=/home/sath/qemu/ppc64-softmmu/qemu-system-ppc64 \
emulator_path=/home/sath/qemu/ppc64-softmmu/qemu-system-ppc64 \
mem=81920 \
smp=512 \
take_regular_screendumps=no backup_image_before_testing=no \
libvirt_controller=virtio-scsi scsi_hba=virtio-scsi-pci drive_format=scsi-hd \
use_os_variant=no \
restore_image_after_testing=no \
vga=none \
display=nographic \
kernel=/home/sath/linux/vmlinux \
kernel_args='root=/dev/sda2 rw console=tty0 console=ttyS0,115200 init=/sbin/init initcall_debug selinux=0' \
virtinstall_qemu_cmdline_vm1=' -M pseries,ic-mode=dual' \
virtinstall_qemu_cmdline_vm2=' -M pseries,ic-mode=xive' \
virtinstall_qemu_cmdline_vm3=' -M pseries,ic-mode=xics' \
virtinstall_qemu_cmdline_vm4=' -M pseries,ic-mode=dual,max-cpu-compat=power8' \
virtinstall_qemu_cmdline_vm5=' -M pseries,ic-mode=dual,kernel-irqchip=off' \
virtinstall_qemu_cmdline_vm6=' -M pseries,ic-mode=xive,kernel-irqchip=off' \
virtinstall_qemu_cmdline_vm7=' -M pseries,ic-mode=xics,kernel-irqchip=off' \
virtinstall_qemu_cmdline_vm8=' -M pseries,ic-mode=dual,max-cpu-compat=power8,kernel-irqchip=off' \
initrd='' \
use_serial_login=yes \
kill_vm=yes kill_vm_libvirt=yes env_cleanup=yes create_vm_libvirt=yes \
vcpu_maxcpus=512 \
test_timeout=1440000 \
vms='vm1 vm2 vm3 vm4 vm5 vm6 vm7 vm8' \
stress_events='reboot' \
stress_itrs=10 \
vcpu_cores_vm1=128 vcpu_threads_vm1=4 \
vcpu_cores_vm2=256 vcpu_threads_vm2=2 \
vcpu_cores_vm3=64 vcpu_threads_vm3=8 \
vcpu_cores_vm4=512 vcpu_threads_vm4=1 \
vcpu_cores_vm5=128 vcpu_threads_vm5=4 \
vcpu_cores_vm6=256 vcpu_threads_vm6=2 \
vcpu_cores_vm7=64 vcpu_threads_vm7=8 \
vcpu_cores_vm8=512 vcpu_threads_vm8=1 \
guest_stress=no

JOB ID     : dcf3a14bae9d4007735a49a250114a6361dcea9f
JOB LOG    : /home/sath/job-2019-06-20T04.52-dcf3a14/job.log
 (1/1) type_specific.io-github-autotest-libvirt.multivm_cpustress.custom_host_events.custom_vm_events:  PASS (1214.78 s)
RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB TIME   : 1216.73 s
JOB HTML   : /home/sath/job-2019-06-20T04.52-dcf3a14/results.html

Explain on this test w.r.t command line parameters:

This test reboots 8 KVM guests booted with given upstream kernel,each with
different interrupt controller and different cpu topology in parallel
for given user iterations, `stress_itrs=10` on given upstream qemu.

If you are interested in KVM Testing, there are much more possibilities you can explore with Avocado-VT and reach us through mailing list for any suggestions or queries.

Hope this blog helps you in someways :-)

]]>Satheesh Rajendransathnaga86@gmail.comRun Syzkaller Using Avocado And Op Test2019-03-19T00:00:00+00:002019-03-19T00:00:00+00:00https://sathnaga86.com/2019/03/19/run-syzkaller-using-avocado-and-op-testThis blog explains how avocado and op-test helps us to setup and run syzkaller on Power Platform

What is syzkaller:

syzkaller is an unsupervised, coverage-guided kernel fuzzer

Avocado test enables support:

Testcase: syzkaller.power

What the testcase do:

  1. Install/Setup syzkaller in host.
  2. Setup Guest passwordless ssh from host.
  3. Prepare and compile Guest kernel.
  4. Prepare syzkaller config with user given qemu params and guest params.
  5. Start sykaller with above config and run for specified time(test_timeout).
  6. Fails out the test in case of host issues.

syzkaller specific testcase config parameters:

Below syzkaller specific test parameters with default values as given here, which can be modified in avocado command line to match as per our needs using --vt-extra-params as other params as mentioned in below avocado command line. More details on the syzkaller config params can be found under here

syz_qemu_args = "-enable-kvm -M pseries -net nic,model=virtio"
syz_kernel_repo = "https://github.com/linuxppc/linux.git"
syz_kernel_branch = "merge"
syz_kernel_config = "ppc64le_guest_defconfig"
syz_target = "linux/ppc64le"
syz_cmd_params = "-debug -v 10"
syz_http = "0.0.0.0:56741"
syz_count = 1
syz_procs = 4

How to use:

ENV Used for this blog:

  • Host : IBM Power8 runs Fedora 28 with kernel 5.0.0
  • Guest: Fedora 27 with kernel 5.1.0-rc1-g3770244b0
  • Qemu: 3.1.50 (v3.1.0-3117-ga9d1cc9f56-dirty)

Below is the sample avocado command line that runs above mentioned syzkaller test

# avocado run syzkaller --vt-type libvirt --vt-extra-params create_vm_libvirt=yes kill_vm_libvirt=yes env_cleanup=yes mem=20480 smp=2 take_regular_screendumps=no test_timeout=4000 backup_image_before_testing=no libvirt_controller=virtio-scsi scsi_hba=virtio-scsi-pci drive_format=scsi-hd use_os_variant=no restore_image_after_testing=no vga=none display=nographic --vt-guest-os JeOS.27.ppc64le
JOB ID     : 78a8a2399ad98bd0e8999f365eff1539009e9351
JOB LOG    : /home/sath/avocado-fvt-wrapper/results/job-2019-03-18T11.58-78a8a23/job.log
 (1/1) powerkvm-qemu.syzkaller.power: PASS (1830.90 s)
RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB TIME   : 1837.14 s

Running syzkaller testcase through op-test:

There are lot of chances we might hit host crash itself, so let’s run above Avocado test using op-test-framework, so that we can capture host console and debug logs during host crash situations.

op-test configuration file:

$cat host_tests.conf
[op-test]
bmc_type=BMC
bmc_ip=x.x.x.x
bmc_username=ADMIN
bmc_password=ADMIN
bmc_usernameipmi=ADMIN
bmc_passwordipmi=ADMIN
host_ip=x.x.x.x
host_user=root
host_password=passw0rd
host_cmd_timeout=5000
host_cmd_file=./tests.conf
host_cmd_resultpath=/home/syzkaller/latest
machine_state=OS

test configuration file:

This expects avocado-vt,avocado is installed already in the system under test, if not please refer this blog to get it installed, How to install avocado-vt

$cat tests.conf
avocado run syzkaller --vt-type libvirt --vt-extra-params create_vm_libvirt=yes kill_vm_libvirt=yes env_cleanup=yes mem=20480 smp=2 take_regular_screendumps=no backup_image_before_testing=no libvirt_controller=virtio-scsi scsi_hba=virtio-scsi-pci drive_format=scsi-hd use_os_variant=no test_timeout=4000 restore_image_after_testing=no vga=none display=nographic --vt-guest-os JeOS.27.ppc64le --job-results-dir /home/syzkaller --job-timeout 4100

Let’s run now syzkaller test using op-test framework with above mentioned configuration files.

./op-test -c host_tests.conf --run testcases.RunHostTest.RunHostTest
...
...
...
[console-expect]#avocado run syzkaller --vt-type libvirt --vt-extra-params create_vm_libvirt=yes kill_vm_libvirt=yes env_cleanup=yes mem=20480 smp=2 take_regular_screendumps=no backup_image_before_testing=no libvirt_controller=virtio-scsi scsi_hba=virtio-scsi-pci drive_format=scsi-hd use_os_variant=no test_timeout=4000 restore_image_after_testing=no vga=none display=nographic --vt-guest-os JeOS.27.ppc64le --job-results-dir /home/syzkaller --job-timeout 4100
avocado run syzkaller --vt-type libvirt --vt-extra-params create_vm_libvirt=yes kill_vm_libvirt=yes env_cleanup=yes mem=20480 smp=2 take_regular_screendumps=no backup_image_befno test_timeout=4000 restore_image_after_testing=no vga=none display=nographic --vt-guest-os JeOS.27.ppc64le --job-results-dir /home/syzkaller --job-timeout 4100
JOB ID     : b9c81856ceddded5b6a90832c35afe5400091a17
JOB LOG    : /home/syzkaller/job-2019-03-19T03.10-b9c8185/job.log
 (1/1) powerkvm-qemu.syzkaller.power: PASS (3998.13 s)
RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB TIME   : 4004.33 s
JOB HTML   : /home/syzkaller/job-2019-03-19T03.10-b9c8185/results.html
[console-expect]#echo $?
echo $?
0
[console-expect]#OK (4030.016s)

----------------------------------------------------------------------
Ran 1 test in 4030.016s

OK

Hope this blog help you in someway, Thanks for reading…

]]>Satheesh Rajendransathnaga86@gmail.comImprove Test Scenarios Using Qemu Code Coverage2019-01-03T00:00:00+00:002019-01-03T00:00:00+00:00https://sathnaga86.com/2019/01/03/improve-test-scenarios-using-qemu-code-coverageBlog on how to measure qemu Code Coverage using avocado-vt framework and use it to improve test scenarios.

What is Code Coverage measurement:

Code coverage is measurement of how many lines of source code blocks/branches are executed during automated test runs and it can be done by instrumenting the source prior using certain tools like gcov and coverage report can be generated using tools like gcovr

How to measure qemu code coverage and view reports for test improvements:

Pre-requisite:

Linux box installed with Avocado framework, you can do referring my previous blogs on how to Install avocado and Running libvirt test using avocado

Environment Used:

Hardware: IBM Power8 Box, Operating system: Fedora29 with kernel 4.20, Qemu: 3.1.50 (v3.0.0-2419-g0e09a75401-dirty), Libvirt: 5.0.0

How to prepare qemu source for Code coverage measurement:

Let’s run below qemu build test to compile qemu binary with code coverage instrumentation(gcov) enabled

# wget https://raw.githubusercontent.com/sathnaga/avocado-vt/kvmci/kvmci/ppc/qemu_build.cfg -O qemu_build.cfg
# avocado run --vt-config qemu_build.cfg --vt-extra-params git_repo_qemu_configure_options="--target-list=ppc64-softmmu --enable-debug --enable-gcov"

Now we have qemu source code/binary instrumented with gcov, lets run a kvm test and measure code covered by the test.

We need the following patch to be merged in avocado-vt to avail the support, Avocado-vt patch to enable qemu code coverage support

For now you can use my kvmci branch for avocado-vt Or directly use above patch in your avocado-vt repository using git am <downloaded patch>.

How to run kvm tests to get code coverage:

Let’s run below vcpu hotplug kvm test to measure code coverage with below params enabled

  • gcov_qemu=yes -> To enable gcov code coverage.
  • gcov_qemu_reset=yes -> Whether or not to erase coverage report generated by previous test.
  • gcov_qemu_compress=yes -> Whether to compress the coverage report, coz with –html-details the coverage report folder can cross few hundreds of MBs.
  • gcov_qemu_collect_cmd_opts=”–exclude-directories=capstone –html –html-details” -> Additional options for code coverage report generation tool(gcovr).
  • qemu_binary,emulator_path=/usr/share/avocado-plugins-vt/build/qemu/ppc64-softmmu/qemu-system-ppc64 -> qemu binary path where previous qemu_build test compiled qemu binary.
# avocado run libvirt_vcpu_plug_unplug.positive_test.vcpu_set.guest.vm_operate.no_operation --vt-type libvirt --vt-extra-params qemu_binary=/usr/share/avocado-plugins-vt/build/qemu/ppc64-softmmu/qemu-system-ppc64 emulator_path=/usr/share/avocado-plugins-vt/build/qemu/ppc64-softmmu/qemu-system-ppc64 create_vm_libvirt=yes kill_vm_libvirt=yes env_cleanup=yes mem=20480 smp=2 take_regular_screendumps=no backup_image_before_testing=no libvirt_controller=virtio-scsi scsi_hba=virtio-scsi-pci drive_format=scsi-hd use_os_variant=no restore_image_after_testing=no vga=none display=nographic kernel=/home/kvmci/linux/vmlinux kernel_args='root=/dev/sda2 rw console=tty0 console=ttyS0,115200 init=/sbin/init initcall_debug selinux=0' gcov_qemu=yes gcov_qemu_reset=yes  gcov_qemu_compress=yes gcov_qemu_collect_cmd_opts="--exclude-directories=capstone --html --html-details" --vt-guest-os JeOS.27.ppc64le --job-results-dir /home/kvmci/
JOB ID     : f196562eb13d65f892b4dd8c941f32d526496dc5
JOB LOG    : /home/kvmci/job-2018-12-18T07.24-f196562/job.log
(1/1) type_specific.io-github-autotest-libvirt.libvirt_vcpu_plug_unplug.positive_test.vcpu_set.guest.vm_operate.no_operation: PASS (169.21 s)
RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB TIME   : 173.94 s
JOB HTML   : /home/kvmci/job-2018-12-18T07.24-f196562/results.html

# ls -l /home/kvmci/job-2018-12-18T07.24-f196562/test-results/1-type_specific.io-github-autotest-libvirt.libvirt_vcpu_plug_unplug.positive_test.vcpu_set.guest.vm_operate.no_operation/
total 11236
drwxr-xr-x 2 root root     4096 Dec 18 07:24 data
-rw-r--r-- 1 root root    38423 Dec 18 07:27 debug.log
-rw-r--r-- 1 root root 11067644 Dec 18 07:27 gcov_qemu.tar.gz
-rw-r--r-- 1 root root    18544 Dec 18 07:27 ip-sniffer.log
...
...

2018-12-18 07:25:46,307 process          L0596 INFO | Running 'rpm -q gcovr'
2018-12-18 07:25:46,325 process          L0428 DEBUG| [stdout] gcovr-4.1-2.fc28.noarch
2018-12-18 07:25:46,326 process          L0684 INFO | Command 'rpm -q gcovr' finished with 0 after 0.0127010345459s
2018-12-18 07:25:46,343 process          L0596 INFO | Running 'gcovr -j 10 -o /home/kvmci/job-2018-12-18T07.24-f196562/test-results/1-type_specific.io-github-autotest-libvirt.libvirt_vcpu_plug_unplug.positive_test.vcpu_set.guest.vm_operate.no_operation/gcov_qemu/gcov.html -s --exclude-directories=capstone --html --html-details .'
2018-12-18 07:27:08,552 process          L0428 DEBUG| [stderr] (WARNING) GCOV produced the following errors processing /usr/share/avocado-plugins-vt/build/qemu/migration/colo-comm.gcno:
2018-12-18 07:27:08,552 process          L0428 DEBUG| [stdout] lines: 12.2% (35112 out of 288554)

Above gcov_qemu.tar.gz contains detailed html code coverage reports and can be extracted and viewed in browser, sample report looks like below

Overall Coverage report:

Individual files line and branch coverage:

Conclusion:

Now we should be able to check what part of code gets exercised while running our individual blackbox avocado tests, using the same we can improve test scenarios to address missing part of code.

Thanks for reading, hope this helps you in someways….

]]>Satheesh Rajendransathnaga86@gmail.comDevops Per Patch Kvmci2018-11-29T00:00:00+00:002018-11-29T00:00:00+00:00https://sathnaga86.com/2018/11/29/devops-per-patch-kvmciPer-patch KVM CI using open source recipes

Used recipes:

snowpatch - Used to listen on upstream patches from given mailing list and trigger the jenkins job.

Jenkins - Used to orchestrate the CI workflow using pipeline.

op-test-framework - Framework used to install upstream kernel and run tests.

Avocado framework - Framework used to run OS, KVM tests.

Hardware used: IBM Power8 box

WorkFlow:

  1. snowpatch is configured to listen on a powerpc kernel mailing list and it does create a local branch with the new patch series on the mailing list and triggers the given jenkins job with the repository and branch details.

    sample snowpatch output screenshot

  2. Jenkins pipeline as shown is configured to run below workflow

  3. Notify Slack on completion

Results can be viewed in Jenkins

Thanks for reading, Hope this helps you in someways…

]]>Satheesh Rajendransathnaga86@gmail.comRun Host Tests Using Op Test Framework2018-11-11T00:00:00+00:002018-11-11T00:00:00+00:00https://sathnaga86.com/2018/11/11/run-host-tests-using-op-test-frameworkRun any host tests on IBM OpenPower boxes using op-test-framework.

Introduction to op-test-framework:

op-test-framework is a python unittest based test suite for validating IBM OpenPower boxes, which comprises many tests including booting host with multiple configurations etc.

Test enables the support:

RunHostTest.py

Why to run other framework tests through op-test-framework:

op-test-framework has it own benefits like

  1. Enable console support for IBM Power boxes
  2. Automatic Runtime failure detection like host crash
  3. Power cycle support
  4. pre-written libraries for various IPMI commands

But we might already have testcases reside in other frameworks,like Avocado. and shortcomings of those framework would be, environment will get lost, when a testcase crashes the host and most of similar tests are designed to run within host.

So instead of rewriting whole test in op-test-framework to enjoy its features. We can just run tests from those frameworks through op-test-framework.

How to use it:

We can use it two ways:

  • run a single command in host(SUT) using option (–host-cmd)
  • run multiple commands in host(SUT) by supplying a file as input where each line is considered as a command and it detects reboot as a special line and takes care of host power cycle.(–host-cmd-file)

Sample configuration file:

$cat host_tests.conf
[op-test]
bmc_type=BMC
bmc_ip=x.x.x.x
bmc_username=ADMIN
bmc_password=ADMIN
bmc_usernameipmi=ADMIN
bmc_passwordipmi=ADMIN
host_ip=x.x.x.x
host_user=root
host_password=passw0rd
host_cmd_timeout=36000
host_cmd_file=./tests.conf
host_cmd_resultpath=/home/kvmci/latest
machine_state=OS

Explanation to above op-test config file:

host and bmc credentials which is common for most of the tests and
host_cmd_timeout - timeout for each command in the file(just given maximum timeout).
host_cmd_file - file path, relative to op-test-framework base directory.
host_cmd_resultpath - folder path inside the host(SUT) to copy back the results.

An example host_cmd_file:

  • compiles upstream guest kernel, boots and runs a KVM guest vcpuhotplug test. 
    $cat tests.conf
[ -d /home/kvmci/ ] || mkdir -p /home/kvmci
echo "KVMCI: Building Upstream guest kernel..."
[ -d /home/kvmci/linux ] || (cd /home/kvmci && git clone --depth 1 https://git.kernel.org/pub/scm/linux/kernel/git/powerpc/linux.git -b merge)
cd /home/kvmci/linux
make ppc64le_guest_defconfig
make -j 240
avocado run libvirt_vcpu_plug_unplug.positive_test.vcpu_set.guest.vm_operate.no_operation --vt-type libvirt --vt-extra-params create_vm_libvirt=yes kill_vm_libvirt=yes env_cleanup=yes mem=20480 smp=2 take_regular_screendumps=no backup_image_before_testing=no libvirt_controller=virtio-scsi scsi_hba=virtio-scsi-pci drive_format=scsi-hd use_os_variant=no restore_image_after_testing=no vga=none display=nographic kernel=/home/kvmci/linux/vmlinux kernel_args='root=/dev/sda2 rw console=tty0 console=ttyS0,115200 init=/sbin/init initcall_debug selinux=0' --vt-guest-os JeOS.27.ppc64le --job-results-dir /home/kvmci/

Let’s run:

$git clone https://github.com/open-power/op-test-framework
$cd op-test-framework
$./op-test --run testcases.RunHostTest.RunHostTest -c ./host_tests.conf

...
...

[console-expect]#avocado run libvirt_vcpu_plug_unplug.positive_test.vcpu_set.guest.vm_operate.no_operation --vt-type libvirt --vt-extra-params create_vm_libvirt=yes kill_vm_libvirt=yes env_cleanup=yes mem=20480 smp=2 take_regular_screendumps=no backup_image_before_testing=no libvirt_controller=virtio-scsi scsi_hba=virtio-scsi-pci drive_format=scsi-hd use_os_variant=no restore_image_after_testing=no vga=none display=nographic kernel=/home/kvmci/linux/vmlinux kernel_args='root=/dev/sda2 rw console=tty0 console=ttyS0,115200 init=/sbin/init initcall_debug selinux=0' --vt-guest-os JeOS.27.ppc64le --job-results-dir /home/kvmci/
<nel=/home/kvmci/linux/vmlinux kernel_args='root=/dev/sda2 rw console=tty0 console=ttyS0,115200 init=/sbin/init initcall_debug selinux=0' --vt-guest-os JeOS.27.ppc64le --job-results-dir /home/kvmci/
JOB ID     : b291eb36bb522ff6a3c4d8db1b9124368e20e2bc
JOB LOG    : /home/kvmci/job-2018-11-09T00.19-b291eb3/job.log
 (1/1) type_specific.io-github-autotest-libvirt.libvirt_vcpu_plug_unplug.positive_test.vcpu_set.guest.vm_operate.no_operation:  PASS (75.76 s)
RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB TIME   : 78.36 s
JOB HTML   : /home/kvmci/job-2018-11-09T00.19-b291eb3/results.html
[console-expect]#echo $?
echo $?
0
OK (113.264s)

----------------------------------------------------------------------
Ran 1 test in 113.265s

OK

Generating XML reports...

References:

Guest kernel config

Avocado KVM Tests

Hope this helps you :-), let’s break the boxes with tests…

]]>Satheesh Rajendransathnaga86@gmail.comUpstream Kernel Install Using Op Test Framework2018-11-08T00:00:00+00:002018-11-08T00:00:00+00:00https://sathnaga86.com/2018/11/08/upstream-kernel-install-using-op-test-frameworkInstall/boot Upstream Kernel on IBM OpenPower boxes using op-test-framework.

Introduction to op-test-framework:

op-test-framework is a python unit-test based test suite for validating IBM OpenPower boxes, which comprises many tests including booting host with multiple configurations etc.

Test enables the support:

InstallUpstreamKernel.py

What does this test do:

This test takes input of upstream(any) kernel git repository, branch and optionally a patch to be applied on top of the given git repository and it builds, installs and boots that kernel into given IBM OpenPower box and makes it ready for further tests on it.

How to use it:

Lets see how to build, install and boot upstream kernel on IBM Power Host.

Clone op-test framework repository in any of the remote system(not the SUT):

$git clone https://github.com/open-power/op-test-framework
$cd op-test-framework

Create a machine.conf to match the host(SUT) that you want to install upstream kernel like below

[op-test]
bmc_type=BMC
bmc_ip=x.x.x.x
bmc_username=ADMIN
bmc_password=ADMIN
bmc_usernameipmi=ADMIN
bmc_passwordipmi=ADMIN
host_ip=x.x.x.x
host_user=root
host_password=passw0rd
host_cmd_timeout=36000
git_repo=https://github.com/linuxppc/linux.git
git_branch=merge
git_repoconfigpath=https://raw.githubusercontent.com/sathnaga/avocado-vt/kvmci/kvmci/ppc/config_kvmppc
git_patch=958675.mbox
git_home=/home/kvmci
machine_state=OS

Explanation of config file:

  • git_repo - Upstream kernel repository to be used.
  • git_branch(optional) - Branch of upstream kernel repository to be used, by default master.
  • git_repoconfigpath(optional) - Kernel configuration can be supplied as url or local file path, by default ppc64le_defconfig will be used.
  • git_patch(optional) - Any incremental patch to be applied on top of above kernel repository, which can be supplied as url or local file path.
  • git_home(optional) - Base path for kernel repository clone in SUT, by default /home/ci will be used.

Running Upstream Kernel Install test…

$./op-test -c machine.conf --run testcases.InstallUpstreamKernel.InstallUpstreamKernel


...

...

[console-expect]#2018-11-08 17:57:10,997:op-test.testcases.InstallUpstreamKernel:runTest:INFO:Installed upstream kernel version: 4.20.0-rc1-g16a0961d6

...

[console-expect]#echo $?
echo $?
0
[console-expect]#OK (836.924s)
----------------------------------------------------------------------
Ran 1 test in 836.924s

OK

Now your server has the upstream kernel installed and ready to test :-) …

Hope this helps, Thanks for reading…

]]>Satheesh Rajendransathnaga86@gmail.comHow to setup Continuous integration test for qemu, libvirt and guest kernel development using Avocado Test Framework2018-05-24T00:00:00+00:002018-05-24T00:00:00+00:00https://sathnaga86.com/2018/05/24/devops-libvirt-qemu-kernel-ci-for-kvmRefer my previous blogs for introductions of avocado test framework.

Below are Commits which enabled support:

  • 74fbe0cf Added support Libvirt package build test.
  • a95d2972 Add qemu binary entry for powerpc.
  • 8aaceb14 Minor fixes for custom boot params.
  • 09d46e3 Added virt-install kernel and initrd params support.
  • ab328a91 Add libvirt build test config.

Host Environment: IBM Power8 runs Linux/Fedora28.

Steps to setup CI on KVM IBM Power Host:

  1. Installing packages needed for qemu and libvirt building: 
     # yum groupinstall "Development tools"
 # yum install pixman-devel libtool rpcgen  libnl3-devel libxml2-devel mingw64-portablexdr libtirpc-devel yajl-devel device-mapper-devel libpciaccess-devel rpm-build audit-libs-devel augeas xfsprogs-devel avahi-devel bash-completion cyrus-sasl-devel dbus-devel fuse-devel glusterfs-api-devel glusterfs-devel gnutls-devel libacl-devel libattr-devel libblkid-devel libcap-ng-devel libcurl-devel libpcap-devel librados2-devel librbd1-devel libssh-devel libssh2-devel libtasn1-devel libwsman-devel ncurses-devel netcf-devel numactl-devel parted-devel readline-devel sanlock-devel scrub systemtap-sdt-devel wireshark-devel jansson-devel libiscsi-devel

Refer my previous blogs for installing avocado framework, lets skip that and jump to test runs.

  1. Building upstream qemu:

     # wget https://raw.githubusercontent.com/sathnaga/avocado-vt/kvmci/kvmci/ppc/qemu_build.cfg -O /home/kvmci/qemu_build.cfg
 # avocado run --vt-config /home/kvmci/qemu_build.cfg
 JOB ID     : ebf827140b57cd2e183b29b372d9e4c81f3e2931
 JOB LOG    : /home/sath/avocado-fvt-wrapper/results/job-2018-05-22T02.37-ebf8271/job.log
 (1/1) build: PASS (22.61 s)
 RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
 JOB TIME   : 33.63 s
 JOB HTML   : /home/sath/avocado-fvt-wrapper/results/job-2018-05-22T02.37-ebf8271/results.html

    The above test will clone build the upstream/given qemu gitrepo in qemu_build.cfg which can be modified as per need and softlink qemu-binary in this path /usr/share/avocado-plugins-vt/bin/qemu.

  2. Building upstream libvirt…

     # wget https://raw.githubusercontent.com/sathnaga/avocado-vt/kvmci/kvmci/ppc/libvirt_build.cfg -O /home/kvmci/libvirt_build.cfg
 # avocado run --vt-config /home/kvmci/libvirt_build.cfg
 JOB ID     : a198d64a434efc67751612bcac27e214267826dc
 JOB LOG    : /home/sath/avocado-fvt-wrapper/results/job-2018-05-21T13.04-a198d64/job.log
 (1/1) build: PASS (517.98 s)
 RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
 JOB TIME   : 746.36 s
 JOB HTML   : /home/sath/avocado-fvt-wrapper/results/job-2018-05-21T13.04-a198d64/results.html

    The above test will clone build the upstream/given libvirt gitrepo in libvirt_build.cfg which can be modified as per need and install/replace the system libvirt binary and start the libvirtd service.

  3. Building upstream Guest VM kernel…

     # [ -d /home/linux ] || (mkdir -p /home/ && cd /home/ && git clone https://github.com/torvalds/linux.git)
 # cd /home/linux
 # make ppc64le_guest_defconfig
 # make -j 240 -s

    The above steps as shown will compile the upstream kernel, the config used here is tweaked to accommodate the modules in built in vmlinux as we do not depend on initramfs for booting the guest and this is tested working with 4.17 kernel.

    # setenforce 0 –> as libvirt will fail to boot custom qemu path due to selinux

  4. Running kvm test with upstream qemu, upstream libvirt and upstream guest kernel…

     # avocado run guestpin.with_emualorpin.sequential.positive.with_cpu_hotplug --vt-type libvirt --vt-extra-params emulator_path=/usr/share/avocado-plugins-vt/bin/qemu create_vm_libvirt=yes kill_vm_libvirt=yes env_cleanup=yes smp=2 backup_image_before_testing=no libvirt_controller=virtio-scsi scsi_hba=virtio-scsi-pci drive_format=scsi-hd use_os_variant=no restore_image_after_testing=no vga=none display=nographic kernel=/home/linux/vmlinux kernel_args='root=/dev/sda2 rw console=tty0 console=ttyS0,115200 init=/sbin/init  initcall_debug' take_regular_screendumps=no --vt-guest-os JeOS.27.ppc64le
 JOB ID     : 569dd1a8acf99002ce4c3ae5d1702b6f28b40428
 JOB LOG    : /home/sath/avocado-fvt-wrapper/results/job-2018-05-23T11.56-569dd1a/job.log
  (1/1) type_specific.io-github-autotest-libvirt.guestpin.with_emualorpin.sequential.positive.with_cpu_hotplug: PASS (108.52 s)
 RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
 JOB TIME   : 119.89 s
 JOB HTML   : /home/sath/avocado-fvt-wrapper/results/job-2018-05-23T11.56-569dd1a/results.html

    The above test uses the artifact(binaries) that got generated as outcome of the previous steps and runs a sample KVM test with upstream qemu, libvirt and guest kernel.

Pre-configured script:

I have written a shell script which does above described steps except dependency package installation, this can be executed on a IBM PowerKVM host and get the KVM CI working!!!

# wget https://raw.githubusercontent.com/sathnaga/avocado-vt/kvmci/kvmci/ppc/kvmci_hostcmd.sh
#./kvmci_hostcmd.sh
KVMCI: Building Upstream Kernel...
KVMCI: Installing avocado...
KVMCI: Installing avocado-vt...
KVMCI: Bootstrapping avocado-vt...
KVMCI: Building Upstream Qemu...
JOB ID     : 7effbff2eb661a4979c58e7891158fd183cd5f95
JOB LOG    : /root/avocado/job-results/job-2018-05-24T06.15-7effbff/job.log
 (1/1) build: PASS (22.50 s)
RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB TIME   : 32.53 s
JOB HTML   : /root/avocado/job-results/job-2018-05-24T06.15-7effbff/results.html
KVMCI: Building Upstream Libvirt...
JOB ID     : 5992e90c9a1ebf4ebd8b53758c986bd2df4fb30b
JOB LOG    : /root/avocado/job-results/job-2018-05-24T06.15-5992e90/job.log
 (1/1) build: PASS (515.30 s)
RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB TIME   : 747.75 s
JOB HTML   : /root/avocado/job-results/job-2018-05-24T06.15-5992e90/results.html
KVMCI: Running tests with Upstream qemu, libvirt, guest kernel...
JOB ID     : 3c21388a07c43c0bc6efdc18ca869c12f8ec4efd
JOB LOG    : /root/avocado/job-results/job-2018-05-24T06.28-3c21388/job.log
 (1/1) type_specific.io-github-autotest-libvirt.guestpin.with_emualorpin.sequential.positive.with_cpu_hotplug: PASS (108.77 s)
RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB TIME   : 125.67 s
JOB HTML   : /root/avocado/job-results/job-2018-05-24T06.28-3c21388/results.html

Hope this blog help you in someway, Thanks for reading…

]]>Satheesh Rajendransathnaga86@gmail.comDeploy OS on OpenPower boxes using op-test-framework2018-05-22T11:29:00+00:002018-05-22T11:29:00+00:00https://sathnaga86.com/2018/05/22/deploy-os-on-openpower-boxes-using-op


This blog explains about my patches to the op-test-framework to improve/enable OS deployment on OpenPower boxes.


Introduction:

op-test-framework is a python unittest based test suite for validating OpenPower boxes,
which comprises many tests including booting host with multiple configurations etc.

Patches that enabled the support:

op-test-framework: *
4edece0 Add support for HostOS Installation
1b581fb Add support for RHEL OS installation
c2a96d4 Add install library and update Ubuntu install test
830a43c Add commandline params needed for OS deploy
403d141 Fallback to alternate method to determine ip for boxes with older nc utility
* Merged in upstream


Steps to deploy OS:

Lets see how to deploy Fedora28 on IBM Power Host.

$git clone https://github.com/open-power/op-test-framework
$cd op-test-framework

create a machine.conf to match the host that you want to deploy like below

[op-test]
bmc_type=OpenBMC
bmc_ip=x.x.x.x
bmc_username=ADMIN
bmc_password=ADMIN
bmc_usernameipmi=ADMIN
bmc_passwordipmi=ADMIN
host_ip=x.x.x.x
host_user=root
host_password=passw0rd
host_gateway=x.x.x.x
host_dns=x.x.x.x
host_mac=98:be:94:06:de:95 -->change as per your host
host_submask=255.255.255.0  -->change as per your host
host_scratch_disk=/dev/disk/by-id/scsi-35000c50098a05d6f  -->change as per your host
os_repo=https://dl.fedoraproject.org/pub/fedora-secondary/releases/28/Server/ppc64le/os/

Running OS deploy test....
$./op-test -c machine.conf --run testcases.InstallRhel.InstallRhel
....
....
^[[32m[   39.218896] ^[[0m^[[33mtg3 0005:09:00.0 net0^[[0m: EEE is disabled
^[[32m[   39.218976] ^[[0m^[[33mIPv6^[[0m: ADDRCONF(NETDEV_CHANGE): net0: link becomes ready
[console-pexpect]#echo $?
0
/home/jenkins_data/sath/op-test-framework/test-reports/Kernel_dmesg_log_Thu_May_17_11:20:20_2018.log
OK (939.294s)

----------------------------------------------------------------------
Ran 1 test in 939.294s

OK

Generating XML reports...
Output written to: /home/jenkins_data/sath/op-test-framework/test-reports/*20180517110439*

Now you have IBM Power system with Fedora28 installed!!



P:S:- osimages/rhel/rhel.ks has the template for kickstart file, you can tune based on the packages that you want to install, for installing fedora make sure java is removed as it does not available in fedora repository.




Hope this helps you in someways!!!!
Thanks for taking time in reading this blog......

]]>Satheesh