OiO.lk Community platform!

Oio.lk is an excellent forum for developers, providing a wide range of resources, discussions, and support for those in the developer community. Join oio.lk today to connect with like-minded professionals, share insights, and stay updated on the latest trends and technologies in the development field.
  You need to log in or register to access the solved answers to this problem.
  • You have reached the maximum number of guest views allowed
  • Please register below to remove this limitation

TensorFlow Extended (TFX): pipeline outputs is empty even though the pipeline is running correctly

  • Thread starter Thread starter Sagnik Taraphdar
  • Start date Start date
S

Sagnik Taraphdar

Guest
The following code simulates a simple pipeline ingesting a CSV file and converting it to TFRecord.

You can also see the corresponding notebook: https://colab.research.google.com/drive/1GEytZjnNZZ7r_f9QQ9FbauohKNLGSooC?usp=sharing

Code:
output_config = example_gen_pb2.Output(split_config=
    example_gen_pb2.SplitConfig(splits=[
        example_gen_pb2.SplitConfig.Split(name='train', hash_buckets=8),
        example_gen_pb2.SplitConfig.Split(name='eval', hash_buckets=2)
    ])
)

example_gen = CsvExampleGen(
    input_base='data',
    output_config=output_config
)

pipeline_root = 'artifacts'

pipeline = Pipeline(
    pipeline_name='testing pipeline',
    pipeline_root=pipeline_root,
    components=[example_gen],
    enable_cache=True,
    metadata_connection_config=metadata.sqlite_metadata_connection_config(
        os.path.join('artifacts', 'metadata.sqlite')
    )
)

LocalDagRunner().run(pipeline)

I have manually verified that the TFRecord have been properly generated. However, the pipeline's outputs dictionaries is empty.

Code:
print(pipeline.outputs)
# output: {}

This problem persists in both a .ipynb notebook and .py file.

Interestingly enough, InteractiveContext does not have this problem.

Does anyone know what is causing this?
<p>The following code simulates a simple pipeline ingesting a CSV file and converting it to TFRecord.</p>
<p>You can also see the corresponding notebook: <a href="https://colab.research.google.com/drive/1GEytZjnNZZ7r_f9QQ9FbauohKNLGSooC?usp=sharing" rel="nofollow noreferrer">https://colab.research.google.com/drive/1GEytZjnNZZ7r_f9QQ9FbauohKNLGSooC?usp=sharing</a></p>
<pre class="lang-py prettyprint-override"><code>output_config = example_gen_pb2.Output(split_config=
example_gen_pb2.SplitConfig(splits=[
example_gen_pb2.SplitConfig.Split(name='train', hash_buckets=8),
example_gen_pb2.SplitConfig.Split(name='eval', hash_buckets=2)
])
)

example_gen = CsvExampleGen(
input_base='data',
output_config=output_config
)

pipeline_root = 'artifacts'

pipeline = Pipeline(
pipeline_name='testing pipeline',
pipeline_root=pipeline_root,
components=[example_gen],
enable_cache=True,
metadata_connection_config=metadata.sqlite_metadata_connection_config(
os.path.join('artifacts', 'metadata.sqlite')
)
)

LocalDagRunner().run(pipeline)
</code></pre>
<p>I have manually verified that the TFRecord have been properly generated. However, the pipeline's outputs dictionaries is empty.</p>
<pre class="lang-py prettyprint-override"><code>print(pipeline.outputs)
# output: {}
</code></pre>
<p>This problem persists in both a .ipynb notebook and .py file.</p>
<p>Interestingly enough, <code>InteractiveContext</code> does not have this problem.</p>
<p>Does anyone know what is causing this?</p>
 
Top