databricksruntime/python:12.2-LTS: jinja2 ImportError

[marcindulak]

On https://docs.databricks.com/en/release-notes/runtime/12.2lts.html, the version of MarkupSafe 2.0.1 is listed, but databricksruntime/python:12.2-LTS uses MarkupSafe 2.1.3. Other runtimes also seem to have this problem (checked for example standard:13.3-LTS).

There appear two issues with this:

1. jinja2 has an ImportError ImportError: cannot import name 'soft_unicode' from 'markupsafe' (/usr/local/lib/python3.8/site-packages/markupsafe/__init__.py) pallets/markupsafe#284

   IMAGE=databricksruntime/python:12.2-LTS
   docker run --rm -it --name databricks  $IMAGE bash -c "/databricks/python3/bin/python --version"
   docker run --rm -it --name databricks  $IMAGE bash -c "/databricks/python3/bin/pip --version"
   docker run --rm -it --name databricks  $IMAGE bash -c "/databricks/python3/bin/pip show Jinja2 MarkupSafe"
   docker run --rm -it --name databricks  $IMAGE bash -c "/databricks/python3/bin/python -c 'from jinja2 import environment'"
   

   Output

   Python 3.9.5
   pip 21.2.4 from /databricks/python3/lib/python3.9/site-packages/pip (python 3.9)
   Name: Jinja2
   Version: 2.11.3
   Summary: A very fast and expressive template engine.
   Home-page: https://palletsprojects.com/p/jinja/
   Author: Armin Ronacher
   Author-email: armin.ronacher@active-4.com
   License: BSD-3-Clause
   Location: /databricks/python3/lib/python3.9/site-packages
   Requires: MarkupSafe
   Required-by: 
   ---
   Name: MarkupSafe
   Version: 2.1.3
   Summary: Safely add untrusted strings to HTML/XML markup.
   Home-page: https://palletsprojects.com/p/markupsafe/
   Author: 
   Author-email: 
   License: BSD-3-Clause
   Location: /databricks/python3/lib/python3.9/site-packages
   Requires: 
   Required-by: Jinja2
   Traceback (most recent call last):
     File "<string>", line 1, in <module>
     File "/databricks/python3/lib/python3.9/site-packages/jinja2/__init__.py", line 12, in <module>
       from .environment import Environment
     File "/databricks/python3/lib/python3.9/site-packages/jinja2/environment.py", line 25, in <module>
       from .defaults import BLOCK_END_STRING
     File "/databricks/python3/lib/python3.9/site-packages/jinja2/defaults.py", line 3, in <module>
       from .filters import FILTERS as DEFAULT_FILTERS  # noqa: F401
     File "/databricks/python3/lib/python3.9/site-packages/jinja2/filters.py", line 13, in <module>
       from markupsafe import soft_unicode
    ImportError: cannot import name 'soft_unicode' from 'markupsafe' (/databricks/python3/lib/python3.9/site-packages/markupsafe/__init__.py)
   

2. the version of MarkupSafe does not correspond to the version listed on https://docs.databricks.com/en/release-notes/runtime/12.2lts.html. Is that page not up-to-date, or there is a mismatch between the actual databricks 12.2-LTS runtime and its docker container?

[xinzhao-db]

yes, this is an issue, will fix.