Fix VR creation issue while creating VM on shared network using PVLAN by nvazquez · Pull Request #3653 · apache/cloudstack

nvazquez · 2019-10-29T13:53:28Z

Description

Fix VR creation issue when deploying VM on shared network using PVLAN.
Tested on KVM + Open vSwitch host

Error:

2019-10-28 13:37:19,217 DEBUG [c.c.a.t.Request] (Work-Job-Executor-3:ctx-522095d0 job-29/job-30 ctx-22bff2f0) (logid:6771277e) Seq 1-4072942913003192458: Received:  { Ans: , MgmtId: 32988167473142, via: 1(ref-trl-218-k-m7-nvazquez-kvm1), Ver: v1, Flags: 10, { StartAnswer, CheckSshAnswer, GetDomRVersionAnswer, NetworkUsageAnswer, Answer, Answer, Answer } }
2019-10-28 13:37:19,224 DEBUG [o.a.c.n.t.AdvancedNetworkTopology] (Work-Job-Executor-3:ctx-522095d0 job-29/job-30 ctx-22bff2f0) (logid:6771277e) SETUP DHCP PVLAN RULES
2019-10-28 13:37:19,228 DEBUG [c.c.n.r.NetworkHelperImpl] (Work-Job-Executor-3:ctx-522095d0 job-29/job-30 ctx-22bff2f0) (logid:6771277e) Router requires upgrade. Unable to send command to router:4, router template version : null, minimal required version : 4.10.0
2019-10-28 13:37:19,229 WARN  [o.a.c.n.t.AdvancedNetworkVisitor] (Work-Job-Executor-3:ctx-522095d0 job-29/job-30 ctx-22bff2f0) (logid:6771277e) Timed Out
com.cloud.exception.ResourceUnavailableException: Resource [VirtualRouter:4] is unreachable: Unable to send command. Router requires upgrade
	at com.cloud.network.router.NetworkHelperImpl.sendCommandsToRouter(NetworkHelperImpl.java:175)
	at org.apache.cloudstack.network.topology.AdvancedNetworkVisitor.visit(AdvancedNetworkVisitor.java:185)
	at com.cloud.network.rules.DhcpPvlanRules.accept(DhcpPvlanRules.java:61)
	at org.apache.cloudstack.network.topology.AdvancedNetworkTopology.setupDhcpForPvlan(AdvancedNetworkTopology.java:131)
	at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeStart(VirtualNetworkApplianceManagerImpl.java:2073)
	at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1146)

Proposed fix:

Move the VR template version insert into domain_router table before iterating through the VR nics to setup PVLAN

Workaround:

Disable the global setting: router.version.check

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)

Screenshots (if appropriate):

How Has This Been Tested?

KVM + Open vSwitch bridging

Create a Shared network providing:
- Primary VLAN ID
- Secondary VLAN ID
Deploy user VM on network

Before this fix: Observe that VR creation fails -> user VM creation fails
After this fix: Successful VM deployment

nvazquez · 2019-10-29T13:54:49Z

@blueorangutan package

blueorangutan · 2019-10-29T13:55:43Z

@nvazquez a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2019-10-29T14:14:37Z

Packaging result: ✖centos6 ✔centos7 ✔debian. JID-331

nvazquez · 2019-10-29T14:19:21Z

@blueorangutan test

blueorangutan · 2019-10-29T14:19:44Z

@nvazquez a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2019-10-30T00:32:17Z

Trillian test result (tid-429)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 35129 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3653-t429-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_router_nics.py
Smoke tests completed. 76 look OK, 1 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_01_VPC_nics_after_destroy	`Failure`	212.43	test_vpc_router_nics.py

ustcweizhou · 2019-10-30T01:48:26Z

code LGTM

yadvr

Lgtm

nvazquez · 2019-11-01T03:18:58Z

@blueorangutan test matrix

blueorangutan · 2019-11-01T03:19:55Z

@nvazquez a Trillian-Jenkins matrix job (centos7 mgmt + xs71, centos7 mgmt + vmware65, centos7 mgmt + kvmcentos7) has been kicked to run smoke tests

blueorangutan · 2019-11-01T12:19:15Z

Trillian test result (tid-440)
Environment: xenserver-71 (x2), Advanced Networking with Mgmt server 7
Total time taken: 30558 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3653-t440-xenserver-71.zip
Intermittent failure detected: /marvin/tests/smoke/test_scale_vm.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_router_nics.py
Smoke tests completed. 75 look OK, 2 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_01_scale_vm	`Failure`	23.82	test_scale_vm.py
test_01_VPC_nics_after_destroy	`Failure`	254.00	test_vpc_router_nics.py

ustcweizhou · 2019-11-01T12:59:57Z

@nvazquez
the following line needs to be added at around line 2085, because guestNetworks is changed.

        _routerDao.persist(router, guestNetworks);

nvazquez · 2019-11-05T14:27:57Z

Thanks @ustcweizhou I have updated the code
@blueorangutan package

blueorangutan · 2019-11-05T15:07:15Z

@nvazquez a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2019-11-05T15:26:55Z

Packaging result: ✖centos6 ✔centos7 ✔debian. JID-338

nvazquez · 2019-11-05T15:51:06Z

@blueorangutan test matrix

blueorangutan · 2019-11-05T16:19:15Z

@nvazquez a Trillian-Jenkins matrix job (centos7 mgmt + xs71, centos7 mgmt + vmware65, centos7 mgmt + kvmcentos7) has been kicked to run smoke tests

blueorangutan · 2019-11-06T00:41:29Z

Trillian test result (tid-449)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 28519 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3653-t449-kvm-centos7.zip
Smoke tests completed. 77 look OK, 0 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File

blueorangutan · 2019-11-06T01:09:11Z

Trillian test result (tid-448)
Environment: xenserver-71 (x2), Advanced Networking with Mgmt server 7
Total time taken: 30029 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3653-t448-xenserver-71.zip
Intermittent failure detected: /marvin/tests/smoke/test_scale_vm.py
Smoke tests completed. 76 look OK, 1 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_01_scale_vm	`Failure`	15.46	test_scale_vm.py

blueorangutan · 2019-11-06T03:08:15Z

Trillian test result (tid-450)
Environment: vmware-65u2 (x2), Advanced Networking with Mgmt server 7
Total time taken: 37265 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3653-t450-vmware-65u2.zip
Intermittent failure detected: /marvin/tests/smoke/test_internal_lb.py
Intermittent failure detected: /marvin/tests/smoke/test_iso.py
Intermittent failure detected: /marvin/tests/smoke/test_network.py
Intermittent failure detected: /marvin/tests/smoke/test_ssvm.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_router_nics.py
Smoke tests completed. 76 look OK, 1 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_02_internallb_roundrobin_1RVPC_3VM_HTTP_port80	`Failure`	401.73	test_internal_lb.py

GabrielBrascher

Code LGTM

nvazquez · 2019-11-19T03:27:22Z

@blueorangutan package

blueorangutan · 2019-11-19T03:28:04Z

@nvazquez a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2019-11-19T03:46:38Z

Packaging result: ✖centos6 ✔centos7 ✔debian. JID-363

nvazquez · 2019-11-19T03:50:21Z

@blueorangutan test matrix

blueorangutan · 2019-11-19T03:51:04Z

@nvazquez a Trillian-Jenkins matrix job (centos7 mgmt + xs71, centos7 mgmt + vmware65, centos7 mgmt + kvmcentos7) has been kicked to run smoke tests

blueorangutan · 2019-11-19T12:46:53Z

Trillian test result (tid-492)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 30267 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3653-t492-kvm-centos7.zip
Smoke tests completed. 77 look OK, 0 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File

nvazquez · 2019-11-20T02:40:22Z

@blueorangutan test centos7 vmware-65u2

blueorangutan · 2019-11-20T02:41:08Z

@nvazquez a Trillian-Jenkins test job (centos7 mgmt + vmware-65u2) has been kicked to run smoke tests

blueorangutan · 2019-11-20T12:19:09Z

Trillian test result (tid-500)
Environment: vmware-65u2 (x2), Advanced Networking with Mgmt server 7
Total time taken: 32954 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3653-t500-vmware-65u2.zip
Smoke tests completed. 77 look OK, 0 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File

…apache#3653) * Fix VR creation issue while creating VM on shared network using PVLAN * Fix update guest networks

Fix VR creation issue while creating VM on shared network using PVLAN

ce1ce89

nvazquez added type:bug component:advanced-networking labels Oct 29, 2019

nvazquez added this to the 4.14.0.0 milestone Oct 29, 2019

nvazquez self-assigned this Oct 29, 2019

nvazquez requested review from PaulAngus, andrijapanicsb, borisstoyanov and yadvr October 29, 2019 13:54

nvazquez modified the milestones: 4.14.0.0, 4.13.1.0 Oct 29, 2019

ustcweizhou approved these changes Oct 30, 2019

View reviewed changes

yadvr approved these changes Oct 30, 2019

View reviewed changes

Fix update guest networks

5832577

DaanHoogland closed this Nov 7, 2019

DaanHoogland reopened this Nov 7, 2019

GabrielBrascher approved these changes Nov 14, 2019

View reviewed changes

nvazquez closed this Nov 19, 2019

nvazquez reopened this Nov 19, 2019

apache deleted a comment from blueorangutan Nov 20, 2019

yadvr merged commit 80abb5a into apache:master Nov 21, 2019

nvazquez deleted the fixvrpvlanissue branch April 6, 2020 14:35

Conversation

nvazquez commented Oct 29, 2019

Description

Types of changes

Screenshots (if appropriate):

How Has This Been Tested?

Uh oh!

nvazquez commented Oct 29, 2019

Uh oh!

blueorangutan commented Oct 29, 2019

Uh oh!

blueorangutan commented Oct 29, 2019

Uh oh!

nvazquez commented Oct 29, 2019

Uh oh!

blueorangutan commented Oct 29, 2019

Uh oh!

blueorangutan commented Oct 30, 2019

Uh oh!

ustcweizhou commented Oct 30, 2019

Uh oh!

yadvr left a comment

Choose a reason for hiding this comment

Uh oh!

nvazquez commented Nov 1, 2019

Uh oh!

blueorangutan commented Nov 1, 2019

Uh oh!

blueorangutan commented Nov 1, 2019

Uh oh!

ustcweizhou commented Nov 1, 2019

Uh oh!

nvazquez commented Nov 5, 2019

Uh oh!

blueorangutan commented Nov 5, 2019

Uh oh!

blueorangutan commented Nov 5, 2019

Uh oh!

nvazquez commented Nov 5, 2019

Uh oh!

blueorangutan commented Nov 5, 2019

Uh oh!

blueorangutan commented Nov 6, 2019

Uh oh!

blueorangutan commented Nov 6, 2019

Uh oh!

blueorangutan commented Nov 6, 2019

Uh oh!

GabrielBrascher left a comment

Choose a reason for hiding this comment

Uh oh!

nvazquez commented Nov 19, 2019

Uh oh!

blueorangutan commented Nov 19, 2019

Uh oh!

blueorangutan commented Nov 19, 2019

Uh oh!

nvazquez commented Nov 19, 2019

Uh oh!

blueorangutan commented Nov 19, 2019

Uh oh!

blueorangutan commented Nov 19, 2019

Uh oh!

nvazquez commented Nov 20, 2019

Uh oh!

blueorangutan commented Nov 20, 2019

Uh oh!

blueorangutan commented Nov 20, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants