Distributed Shell

Old: Project Goals

• "Toil" for multi-cloud distributed builds: http://www.oilshell.org/blog/2020/11/fixes-and-updates.html#buildssrht-and-toil.
  • Results: http://travis-ci.oilshell.org/
  • Problems:
  • Does too much work (not incremental), and doesn't do it fast enough (not parallel)
    • needs dependencies for both problems!
  • In some cases, the framework has more overhead than the work done by the application. We want lightweight distributed processes.
  • YAML is a really bad syntax for a shell script.
• Pash and Posh are related: https://github.com/oilshell/oil/issues/867

Notes on gg (Ad Hoc Multi-Cloud Distribution with Lambdas)

• Great intro blog post, concentrating mostly on the C++ build use case, which indeed has some unique elements: https://buttondown.email/nelhage/archive/papers-i-love-gg/
  • reaction: distcc pump is another solution to the preprocessor problem, although neither model substitution or distcc pump are fully general
• Great Usenix ATC '19 Video: https://www.youtube.com/watch?v=Cc_MVldSijA&ab_channel=USENIX
• HN comments from July 2019: https://news.ycombinator.com/item?id=20433315
• My initial reaction: https://lobste.rs/s/virbxa/papers_i_love_gg#c_nbmnod
• More reactions
  • I really like the framing: low latency (which is why I use shell in the first place), warm vs. cold clusters
  • IR is extremely similar to Blaze/Forge (and very tiny!)
  • Lambda still has some limitations for huge packages. Good experience report here: https://news.ycombinator.com/item?id=20433315 (sounds like he needs proper dependencies)
  • What about state?
• Concepts
  • Model Substitution
  • Tail Calls
  • Dynamic dependendencies, not static (how does it relate to Shake?)
  • Lambdas can talk to each other (via NAT traversal?) Solves a well known performance issue.
• Citations
  • UCop
  • Ciel

Project Ideas

• Well first, try gg to see how well it works...
• Really basic:
  • Oil can create CLI descriptions for "model substitution"
• Second: Oil front end rather (on top of model sub, "scripting", Python, C++). Does that make sense?
• Run Toil on gg ! For better continuous builds