TensorFlow Meta and Base Object Discrepancies

[brandonwillard]

The connection between meta objects and their names is not one-to-one with the TF graphs in which meta objects create their corresponding reified base objects.

For example, consider the following:

import tensorflow as tf

from tensorflow.python.eager.context import graph_mode

from symbolic_pymc.tensorflow.meta import mt, TFlowMetaNodeDef, TFlowMetaOp


with graph_mode(), tf.Graph().as_default() as test_graph:

    add_tf = tf.add(1, 1)

    assert add_tf.name == 'Add:0'

with graph_mode(), test_graph.as_default():

    # This could be a meta version of `add_tf.op.node_def`
    node_def_mt = TFlowMetaNodeDef('Add', 'Add', {})

    assert mt(add_tf.op.node_def) == node_def_mt

    # This could be a meta version of `add_tf.op`
    inputs_mt = mt(add_tf.op.inputs)
    test_op_mt = TFlowMetaOp(mt.Add, node_def_mt, inputs_mt)

    assert mt(add_tf.op) == test_op_mt

In this example, we've created a TF Add node, add_tf, and created—by hand—a meta NodeDef and meta Operation that could be meta versions of add_tf's constituent Operation and NodeDef base objects.

Now, when we reify the meta Operation, test_op_mt, we find that the base object it produces does not correspond exactly to the underlying meta object, because its name values are different.

with graph_mode(), test_graph.as_default():

    test_op_tf = test_op_mt.reify()

    # The reified, base/TF `Operation` object has a different name!
    assert test_op_mt.name != test_op_tf.name

This discrepancy is due to the fact that test_graph already has a node with the name Add (i.e. the original add_tf's Operation).

One solution: we could check for elements with the desired/specified meta object names in the current TF graph and verify that the proposed meta objects are in-line with those—and, say, raise exceptions when they aren't. At the very least, such a process could save some effort (re)constructing TF objects that already exist and/or better imputation of unspecified information in meta objects (e.g. dtypes and shapes).

Also, TFlowMetaNodeDef needs a custom reify method; the base class method currently does not work for this type.

Originally posted by @brandonwillard in #74 (comment)

[brandonwillard]

Existing names have been checked since #90.

However, an issue arises when reifying meta tensors that are intended to be new, because an unspecified name value is currently set to the default OpDef name (i.e. the operator name). This issue should be easily fixed by using the standard TF means of generating a unique name. I'll put in a fix ASAP.