Current ShinyApp implementation shares the single SparkContext (sc) in one ShinyApp as sc is only destroyed at the end of calling sc.stop(). Therefore, it is not a good practice for resource utilization.
Proposed solution:
The lifecycle of sc should be only one action. Create sc only when spark is needed. After spark operation finished, then terminate directly the SparkContext using spark_disconnect(sc).
Current ShinyApp implementation shares the single SparkContext (sc) in one ShinyApp as sc is only destroyed at the end of calling sc.stop(). Therefore, it is not a good practice for resource utilization.
Proposed solution:
The lifecycle of sc should be only one action. Create sc only when spark is needed. After spark operation finished, then terminate directly the SparkContext using spark_disconnect(sc).