apache_beam.testing.benchmarks.nexmark.nexmark_launcher module

Nexmark launcher.

The Nexmark suite is a series of queries (streaming pipelines) performed on a simulation of auction events. The launcher orchestrates the generation and parsing of streaming events and the running of queries.

Model

• Person: Author of an auction or a bid.

• Auction: Item under auction.

• Bid: A bid for an item under auction.

Events

• Create Person

• Create Auction

• Create Bid

Queries

• Query0: Pass through (send and receive auction events).

Usage

• DirectRunner

  python nexmark_launcher.py –query/q <query number> –project <project id> –loglevel=DEBUG (optional) –wait_until_finish_duration <time_in_ms> –streaming

• DataflowRunner

  python nexmark_launcher.py –query/q <query number> –project <project id> –region <GCE region> –loglevel=DEBUG (optional) –wait_until_finish_duration <time_in_ms> –streaming –sdk_location <apache_beam tar.gz> –staging_location=gs://… –temp_location=gs://

class apache_beam.testing.benchmarks.nexmark.nexmark_launcher.NexmarkLauncher

Bases: object

DONE_DELAY = 300

PERF_DELAY = 20

TERMINATE_DELAY = 3600

WARNING_DELAY = 600

parse_args()

generate_events()

read_from_file()

read_from_pubsub()

run_query(query_num, query, query_args, pipeline_options, query_errors)

monitor(job, event_monitor, result_monitor)

keep monitoring the performance and progress of running job and cancel the job if the job is stuck or seems to have finished running

Returns:

the final performance if it is measured

static log_performance(perf: NexmarkPerf) → None

publish_performance_influxdb(query_num, perf)

static get_performance(result, event_monitor, result_monitor)

cleanup()

run()