Stabilize buffer API

[Tyriar]

Introduced in #2074, we should remove experimental after we know confirm no changes are needed for the search addon to function.

[jerch]

@Tyriar Made some progress in that field in #2369 together with @JavaCS3, see this comment #2369 (comment) and my changes here: master...jerch:serialize_with_private#diff-3cf93a85b230e197db3cdf45b888018f

I am sure you not gonna like the changes much, basically it seems we should still avoid getters.

[Tyriar]

@jerch this blocks the v4 release as we should stabilize such that we don't run into issues with the search addon having broken API in a later v4 release. How about instead converting all the getters into a function we preemptively fetch all the line's values and expose them as regular properties. It's readonly on the API and doesn't matter much if people try to workaround it anyway.

So instead of this:

return new BufferCellApiView(this._line, x);

...

class BufferCellApiView implements IBufferCellApi {
  constructor(private _line: IBufferLine, private _x: number) {}
  public get char(): string { return this._line.getString(this._x); }
  public get width(): number { return this._line.getWidth(this._x); }
}

We do this instead:

return {
    char: this._line.getString(x),
    width: this._line.getWidth(x)
}

Or this:

return new BufferCellApiView(this._line, x);

...


class BufferCellApiView implements IBufferCellApi {
  public char: string;
  public width: number;
  constructor(line: IBufferLine, x: number) {
    this.char = line.getString(x);
    this.width = line.getWidth(x);
  }
}

Reasons I don't think we should do all functions:

• We would be breaking existing (experimental) API, that would result in the search addon for example breaking for all future versions. While we can do this, it is a bit of a pain, plus VS Code uses the buffer API.
• Functions are uglier to use than properties, it would be best if we would be consistent across the API ( IBufferApi.cursorY, IMarker.line, Terminal.element, etc.).
• While there will always be faster ways, they aren't as ergonomic and there are ways around it (don't serialize as much, support incremental serialization, etc.).

/cc @JavaCS3

[jerch]

@Tyriar

You want to accept an API slowdown of 4 - 10 times for the sake of writing cell.xy instead of cell.getXY(), seriously? This is not about something being slightly worse in runtime, this is magnitudes slower for use cases that potentially run quite often and will create hiccups (as any outer custom functionality based on buffer data will have to go through that layer).

A major version jump is the only right place to address things like that, if we miss it now its not for a better xterm.js because ppl will jump on that API and we prolly cannot undo it later on easily. You already argue in that direction

We would be breaking existing (experimental) API ...

Do you think this will get any better if done later?

[Tyriar]

AFAICT this would be the first inconsistency for properties being exposed as a function (except for maybe set/getOption but they have been around forever), it would be a shame if we stopped following the standard now and made it inconsistent as I've been told from several people about how xterm.js is easy to use.

I looked over #2369 again and thought about it, how about an optional argument that needs to be passed to calculate the flags/colors?

interface IBufferLine {
  /**
   * Gets a cell from the line, or undefined if the line index does not exist.
   *
   * Note that the result of this function should be used immediately after
   * calling as when the terminal updates it could lead to unexpected
   * behavior.
   * 
   * @param x The character index to get.
   * @param includeAttributes Optional boolean that indicates whether to
   * calculate and include attribute data (font style, colors) in the returned
   * object.
   * @param cell Optional cell object to recycle as the result of this call.
   * This can boost performance by reducing browser garbage collection time when
   * iterating over the entire buffer.
   */
  getCell(x: number, includeAttributes?: boolean, cell?: IBufferCell): IBufferCell | undefined;
}

interface IBufferCell {
  /**
   * The character within the cell.
   */
  readonly char: string;

  /**
   * The width of the character. Some examples:
   *
   * - This is `1` for most cells.
   * - This is `2` for wide character like CJK glyphs.
   * - This is `0` for cells immediately following cells with a width of `2`.
   */
  readonly width: number;

  /**
   * Text attribute flags like bold, underline etc.
   * 
   * Note that this should only be used when `includeAttributes` is passed to
   * `getCell`. Never store this object as it may get recycled by the library.
   */
  readonly flags: IBufferCellFlags;

  /**
   * Foreground color.
   * 
   * Note that this should only be used when `includeAttributes` is passed to
   * `getCell`. Never store this object as it may get recycled by the library.
   */
  readonly fg: IBufferCellColor;

  /**
   * Background color.
   * 
   * Note that this should only be used when `includeAttributes` is passed to
   * `getCell`. Never store this object as it may get recycled by the library.
   */
  readonly bg: IBufferCellColor;
}

So implementation would do this:

• With the optional cell? as in Add foreground/background color support for SerializeAddon #2369 we can keep GC down.
• The same recycling occurs in flags, fg and bg.
• Only when includeAttributes? is true will flags, fg or bg be valid, when false a shares invalid object could be used which throws with a good error message, for example (this object would be a shared constant):

  const invalidFg = Object.freeze({
    colorMode: () => throw new Error('You must call getCell with includeAttributes to use the fg property'),
    ...
  });
  fg = invalidFg;

This would allow us to keep the naming scheme as well as probably boost performance for cases like #2369 where we need all cell properties since we're dealing with simple object references/primitives rather than in-between functions.

[JavaCS3]

@Tyriar This one looks better since x is a constant value in a sense

class BufferCellApiView implements IBufferCellApi {
  public char: string;
  public width: number;
  constructor(line: IBufferLine, x: number) {
    this.char = line.getString(x);
    this.width = line.getWidth(x);
  }
}

[jerch]

@Tyriar How would the implementation look for the (yet to come) attributes with this? I played with a getter based variant in my playground branch: https://github.com/jerch/xterm.js/blob/09de17dbdb682ed3dbfac7dc3e504f1c28d89055/src/public/Terminal.ts#L238-L281

This variant tries to omit the recreation of subobjects for flags, fg and bg by closuring the .cell property and applying getters on that if a reference cell was given (otherwise all has to be created newly). Yet this shows the weird deopt problem. That was the point for me to say - ok, screw that approach, go with a much simpler linear thingy with access methods.

Also what @JavaCS3 points to might be a source of inconsistency - we have to decide whether a cell in the API should have stable content once requested (created/realized in memory) or always derive from line[x] offset (which might change content between write calls). We have discussed that already in an older thread, but kinda never resolved. If we stick with the line.getString(x) approach (deriving from line[x] offset), we have to extend the internal BufferLine API as well for the attributes to allow "hot loading" of the values.

Yepp, sticking with the optional cell argument will lift a big part of the runtime penalty already, still not sure how the includeAttributes flag will help here? Doesnt that shortcut the attribute processing, thus making any attempt with attributes expensive again? (Well this certainly depends on the actual implementation of the API attributes layer)

Edit: Maybe we should not try to to get an attribute API with v4 yet. To me this still seems to be to unreliable to be spec'ed out. How about just leaving it as it is for now (with experimental tag to indicate possible changes in the future)?

[Tyriar]

still not sure how the includeAttributes flag will help here? Doesnt that shortcut the attribute processing, thus making any attempt with attributes expensive again?

No, the idea was that it would work something like this:

const errorProp = () => throw new Error('You must call getCell with includeAttributes to use this');
// Pretty sure this doesn't compile but it gets the idea across
const invalidColor = { get colorMode(): errorProp, get color(): errorProp, get rgb(): errorProp};
const invalidFlags = ...;
...

constructor(x: number, includeAttributes?: boolean, cell?: IBufferCell) {
  cell = cell || ...;
  if (includeAttributes) {
    // Calculate and set attributes because the user wants them
  } else {
    cell.fg = invalidColor;
    cell.bg = invalidColor;
    cell.flags = invalidFlags;
  }
}

Any call to !includeAttributes would throw, so it shouldn't cause a deopt since cell always has a valid object of the same shape. I believe this I think we might even be able to use conditional types to error at compile time if we wanted to go down that route, especially if we run into problems getting rid of the deopt when throwing the exception.

Maybe we should not try to to get an attribute API with v4 yet.

Sounds better than a hasty decision 👍