Documentation theme and markdown parser

[krassowski]

Two topics to discuss:

• should we switch to the PyData Sphinx Theme which is already used in Jupyter documentation of multiple core projects (but not JupyterLab), as well as pandas, numpy etc?
• should we switch away from recommonmark which does not seem to receive much attention nowadays to MyST parser and MyST-NB to fix issues with Markdown parsing in documentation and get built-in variable substitution with Jinja2 syntax? A few examples of current issues:

Changelog bug fixes and features at the same level (ignoring indent):

Block quotes after code blocks are merged with preceding code block:

Code blocks malformed inside lists:

TBD:

• Should we use substitutions, or wait until the MyST-NB matures enough to support markdown from cell outputs properly (Handle headers in code-cell generated Markdown  executablebooks/MyST-NB#317)?

[bollwyvl]

I'm +:100: on going to myst-nb and pydata-sphinx-theme. jupyter-book would be lovely, but currently has some issues in the packaging department. I also still owe us an icon/wordmark... pst sites kind of anticipate you have a wide one, but it's still tasty to have an icon (favicon, etc).

Additionally on the docs front: our architecture diagrams are getting pretty out-of-date. Updating the grpahviz wouldn't be a huge lift, but ipydrawio is basically ready to rock for these kinds of tasks, with the editable SVG format being the strongest candidate for docs, I'd think, as they look good in github diffs.

[krassowski]

Working on it this weekend.

[bollwyvl]

Very cool! I've actually just been converting a bunch of stuff, and have been quite pleased with the results. Myst-nb is also definitely worth it, and I found it relatively easy to convert notebook metadata programmatically, as I have a similar notebook linter in place. Let me know how I can help!

[chrisjsewell]

Any feedback for MyST welcome, I'd like to eventually look into writing an LSP with this package for MyST in JupyterLab 😄

[bollwyvl]

A myst-ls would be lovely. Pygls is for sure the way to start. Some thoughts: The biggest challenge in adapting an existing parser is that code-as-it's-being-typed is broken 90% of the time, and many parsers (rightly) are expecting well-formed content. Having the docs propagate up would provides a remarkable experience... Did this directive take args or not? A challenge inherited from rst will be, well, rst, and potentially a whole sphinx site for references/includes. A final money piece would be spelling and grammar checking, both of which require a lot of dependencies and configuration to be fit for purpose.

[chrisjsewell]

FYI, I've already dabbled with an RST language server: https://github.com/chrisjsewell/rst-language-server