How do I export my Dataflow worker logs to Cloud Pubsub?

Question

I'd like to run some secondary processing on my worker logs from Cloud Dataflow. Is there an easy way to do that?

Answer 1

This can be done by configuring logs export from Cloud Logging.

First, navigate to the Exports table in the Cloud Logging UI:

On the Exports page, follow these steps:

1. Select the Dataflow service.
2. Deselect All sources.
3. Click Add item and select dataflow.googleapis.com/worker.
4. Select a Pubsub topic to publish to.