[WIP] Add Datera storage plugin by jcsdatera · Pull Request #3410 · apache/cloudstack

jcsdatera · 2019-06-18T06:28:37Z

Features:

Zone-wide and cluster-wide primary storage support
VM template caching automatically on Datera, the subsequent VMs can be created instantaneously by fast cloning the root volume.
Rapid storage-native snapshot
Multiple managed primary storages can be created with a single Datera cluster to provide better management of
- Total provisioned capacity
- Default storage QoS values
- Replica size ( 1 to 5 )
- IP pool assignment for iSCSI target
- Volume Placement ( hybrid, single_flash, all_flash )
Volume snapshot to VM template
Volume to VM template
Volume size increase using service policy
Volume QoS change using service policy
Enabled KVM support
New Datera app_instance name format to include ACS volume name
VM live migration

Description

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)

Screenshots (if appropriate):

How Has This Been Tested?

Features: 1. Zone-wide and cluster-wide primary storage support 2. VM template caching automatically on Datera, the subsequent VMs can be created instantaneously by fast cloning the root volume. 3. Rapid storage-native snapshot 4. Multiple managed primary storages can be created with a single Datera cluster to provide better management of - Total provisioned capacity - Default storage QoS values - Replica size ( 1 to 5 ) - IP pool assignment for iSCSI target - Volume Placement ( hybrid, single_flash, all_flash ) 5. Volume snapshot to VM template 6. Volume to VM template 7. Volume size increase using service policy 8. Volume QoS change using service policy 9. Enabled KVM support 10. New Datera app_instance name format to include ACS volume name 11. VM live migration

swill · 2019-06-19T20:53:50Z

Hey @rhtyd, can we try to get this PR into the 4.13 branch before the RC is cut?

Let us know if there are any outstanding items needed for this to be included. Thanks...

yadvr · 2019-06-20T07:25:07Z

@swill it depends. Is this PR code complete and ready for review.

If this is ready for review we can kick regression smoke tests and merge this on master as soon as this meets our merging guideline (2 lgtms, no regressions).

yadvr

@jcsdatera @swill the added plugin is not loaded in client/pom.xml, so the built jar won't make into cloudstack uber/fat jar, please check if this is intended?

yadvr · 2019-06-25T16:28:25Z

ping @jcsdatera @swill ?

Features: 1. Zone-wide and cluster-wide primary storage support 2. VM template caching automatically on Datera, the subsequent VMs can be created instantaneously by fast cloning the root volume. 3. Rapid storage-native snapshot 4. Multiple managed primary storages can be created with a single Datera cluster to provide better management of - Total provisioned capacity - Default storage QoS values - Replica size ( 1 to 5 ) - IP pool assignment for iSCSI target - Volume Placement ( hybrid, single_flash, all_flash ) 5. Volume snapshot to VM template 6. Volume to VM template 7. Volume size increase using service policy 8. Volume QoS change using service policy 9. Enabled KVM support 10. New Datera app_instance name format to include ACS volume name 11. VM live migration

… into master.datera

Features: 1. Zone-wide and cluster-wide primary storage support 2. VM template caching automatically on Datera, the subsequent VMs can be created instantaneously by fast cloning the root volume. 3. Rapid storage-native snapshot 4. Multiple managed primary storages can be created with a single Datera cluster to provide better management of - Total provisioned capacity - Default storage QoS values - Replica size ( 1 to 5 ) - IP pool assignment for iSCSI target - Volume Placement ( hybrid, single_flash, all_flash ) 5. Volume snapshot to VM template 6. Volume to VM template 7. Volume size increase using service policy 8. Volume QoS change using service policy 9. Enabled KVM support 10. New Datera app_instance name format to include ACS volume name 11. VM live migration

… into master.datera

jcsdatera · 2019-06-26T08:13:26Z

Thank you @rhtyd, I have updated client/pom.xml.

blueorangutan · 2019-06-26T11:12:14Z

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2019-06-26T12:40:25Z

Packaging result: ✖centos6 ✔centos7 ✔debian. JID-41

yadvr · 2019-06-26T12:41:19Z

@blueorangutan package

blueorangutan · 2019-06-26T12:42:13Z

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2019-06-26T13:02:23Z

Packaging result: ✖centos6 ✔centos7 ✔debian. JID-45

yadvr · 2019-06-26T18:46:42Z

@blueorangutan package

blueorangutan · 2019-06-26T18:47:15Z

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2019-06-26T19:07:58Z

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-46

yadvr · 2019-06-27T03:56:58Z

@blueorangutan test

blueorangutan · 2019-06-27T03:57:17Z

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

yadvr · 2019-06-28T06:20:53Z

@jcsdatera @swill the PR fails to start the management server with the error:

2019-06-27 04:08:36,762 WARN  [o.a.c.s.m.c.ResourceApplicationContext] (main:null) (logid:) Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'org.apache.cloudstack.spring.lifecycle.ConfigDepotLifeCycle#0': Unsatisfied dependency expressed through field 'configDepotAdmin'; nested exception is org.springframework.beans.factory.CannotLoadBeanClassException: Cannot find class [org.apache.cloudstack.storage.datastore.provider.DateraPrimaryDataStoreProvider] for bean with name 'dateraDataStoreProvider' defined in URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.13.0.0-SNAPSHOT.jar!/META-INF/cloudstack/storage-volume-datera/spring-storage-volume-datera-context.xml]; nested exception is java.lang.ClassNotFoundException: org.apache.cloudstack.storage.datastore.provider.DateraPrimaryDataStoreProvider
2019-06-27 04:08:36,762 WARN  [o.e.j.w.WebAppContext] (main:null) (logid:) Failed startup of context o.e.j.w.WebAppContext@646be2c3{/client,file:///usr/share/cloudstack-management/webapp/,UNAVAILABLE}{/usr/share/cloudstack-management/webapp}
org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'org.apache.cloudstack.spring.lifecycle.ConfigDepotLifeCycle#0': Unsatisfied dependency expressed through field 'configDepotAdmin'; nested exception is org.springframework.beans.factory.CannotLoadBeanClassException: Cannot find class [org.apache.cloudstack.storage.datastore.provider.DateraPrimaryDataStoreProvider] for bean with name 'dateraDataStoreProvider' defined in URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.13.0.0-SNAPSHOT.jar!/META-INF/cloudstack/storage-volume-datera/spring-storage-volume-datera-context.xml]; nested exception is java.lang.ClassNotFoundException: org.apache.cloudstack.storage.datastore.provider.DateraPrimaryDataStoreProvider
        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:586)
        at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:91)
        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:372)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1344)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:582)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:502)
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:312)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:228)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:310)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:205)
        at org.springframework.context.support.PostProcessorRegistrationDelegate.registerBeanPostProcessors(PostProcessorRegistrationDelegate.java:238)
        at org.springframework.context.support.AbstractApplicationContext.registerBeanPostProcessors(AbstractApplicationContext.java:709)
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:534)
...
Caused by: org.springframework.beans.factory.CannotLoadBeanClassException: Cannot find class [org.apache.cloudstack.storage.datastore.provider.DateraPrimaryDataStoreProvider] for bean with name 'dateraDataStoreProvider' defined in URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.13.0.0-SNAPSHOT.jar!/META-INF/cloudstack/storage-volume-datera/spring-storage-volume-datera-context.xml]; nested exception is java.lang.ClassNotFoundException: org.apache.cloudstack.storage.datastore.provider.DateraPrimaryDataStoreProvider
        at org.springframework.beans.factory.support.AbstractBeanFactory.resolveBeanClass(AbstractBeanFactory.java:1380)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.determineTargetType(AbstractAutowireCapableBeanFactory.java:670)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.predictBeanType(AbstractAutowireCapableBeanFactory.java:637)
        ... 63 more
Caused by: java.lang.ClassNotFoundException: org.apache.cloudstack.storage.datastore.provider.DateraPrimaryDataStoreProvider
        at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)

Can you test and confirm the PR work against master? I'll rekick tests one more time.

@blueorangutan package

blueorangutan · 2019-06-28T06:21:13Z

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

yadvr · 2019-06-28T06:22:58Z

@jcsdatera @swill also check Travis, the jobs are failing there with same errors.

blueorangutan · 2019-06-28T07:25:27Z

Packaging result: ✔centos6 ✖centos7 ✔debian. JID-50

yadvr · 2019-07-03T06:38:28Z

Ping @jcsdatera @swill - if you'd like this to make into 4.13.0.0 please work with me in getting the build/run errors fixed. Thanks.

jcsdatera · 2019-07-05T01:02:52Z

Ping @rhtyd @swill please use PR-3470 instead of this one.

Thank you

yadvr · 2019-07-05T08:39:39Z

Closing this one based on your remark @manojkverma (assuming you're a colleague). cc @swill @jcsdatera

yadvr added the status:work-in-progress label Jun 18, 2019

jcsdatera changed the title ~~[WIP] Add Datera storage plugin~~ Add Datera storage plugin Jun 19, 2019

yadvr mentioned this pull request Jun 20, 2019

[WIP] Add Datera storage plugin #3324

Closed

4 tasks

yadvr added this to the 4.13.0.0 milestone Jun 20, 2019

yadvr requested changes Jun 20, 2019

View reviewed changes

jcsdatera added 6 commits June 26, 2019 08:30

Merge branch 'master.datera' of https://github.com/jcsdatera/cloudstack…

80838bf

… into master.datera

Merge branch 'master.datera' of https://github.com/jcsdatera/cloudstack…

27e15ea

… into master.datera

yadvr changed the base branch from master to 4.11 June 26, 2019 11:09

yadvr changed the base branch from 4.11 to master June 26, 2019 11:09

yadvr removed the status:work-in-progress label Jun 26, 2019

yadvr closed this Jun 26, 2019

yadvr reopened this Jun 26, 2019

yadvr added component:primary-storage type:new-feature labels Jun 26, 2019

yadvr added the status:On Hold label Jul 3, 2019

yadvr changed the title ~~Add Datera storage plugin~~ [WIP] Add Datera storage plugin Jul 3, 2019

yadvr added the status:work-in-progress label Jul 3, 2019

yadvr closed this Jul 5, 2019

Conversation

jcsdatera commented Jun 18, 2019

Description

Types of changes

Screenshots (if appropriate):

How Has This Been Tested?

Uh oh!

swill commented Jun 19, 2019

Uh oh!

yadvr commented Jun 20, 2019

Uh oh!

yadvr left a comment

Choose a reason for hiding this comment

Uh oh!

yadvr commented Jun 25, 2019

Uh oh!

jcsdatera commented Jun 26, 2019

Uh oh!

blueorangutan commented Jun 26, 2019

Uh oh!

blueorangutan commented Jun 26, 2019

Uh oh!

yadvr commented Jun 26, 2019

Uh oh!

blueorangutan commented Jun 26, 2019

Uh oh!

blueorangutan commented Jun 26, 2019

Uh oh!

yadvr commented Jun 26, 2019

Uh oh!

blueorangutan commented Jun 26, 2019

Uh oh!

blueorangutan commented Jun 26, 2019

Uh oh!

yadvr commented Jun 27, 2019

Uh oh!

blueorangutan commented Jun 27, 2019

Uh oh!

yadvr commented Jun 28, 2019

Uh oh!

blueorangutan commented Jun 28, 2019

Uh oh!

yadvr commented Jun 28, 2019

Uh oh!

blueorangutan commented Jun 28, 2019

Uh oh!

yadvr commented Jul 3, 2019

Uh oh!

jcsdatera commented Jul 5, 2019

Uh oh!

yadvr commented Jul 5, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants