druid ingest executor

[rohithreddykota]

Summary

This PR introduces a druidIndexExecutor which is responsible for executing Druid index tasks using configurations from YAML files. The executor handles the ingestion of data from an object store into Druid, calculating necessary intervals, generating dynamic index JSON, and performing the ingestion process based on the model input and output configurations.

Example:

kind: model

refresh:
  cron: 2 * * * *

connector: gcs
path: gs://hitech.rilldata.com/data-export/etl/bids/monthly/
pattern: 'yyyy=2006/mm=01/dd=02/HH=15'
gran: 1h
format: parquet
file_pattern: '.*\.parquet'
retry_period: 15m
max_retires: 3

incremental: true
output:
  connector: druid
  dataSource: demand_log_qa
  initial_look_back_period: 3h
  period_before: 1h
  quiet_period: 1h
  catchup: false
  max_work: 2h # maximum interval to index at a time
  coordinator_url: https://druid.ws1.hitech.rilldata.com/druid/coordinator/v1/datasources
  datasource_name: demand_log
  spec_json: >
    {
      "type": "index_parallel",
      "spec": {
        "ioConfig": {
          "type": "index_parallel",
          "inputSource": {
            "type": "google",
            "prefixes": {{ .prefixes | toJson }},
          },
          "inputFormat": {
            "type": "csv",
            "findColumnsFromHeader": true
          }
        },
        "tuningConfig": {
          "type": "index_parallel",
          "partitionsSpec": {
            "type": "dynamic"
          }
        },
        "dataSchema": {
          "dataSource": "%s",
          "timestampSpec": {
            "column": "timestamp",
            "format": "iso"
          },
          "transformSpec": {},
          "dimensionsSpec": {
            "dimensions": [
              {"type": "long", "name": "id"},
              "publisher",
              "domain",
              {"type": "double", "name": "bid_price"}
            ]
          },
          "granularitySpec": {
            "queryGranularity": "none",
            "rollup": false,
            "segmentGranularity": "day",
            "intervals": {{ .intervals | toJson }}
          }
        }
      }
    }

[begelundmuller]

Some high-level questions:

1. I didn't go deep on the time manipulation logic here, but it seems like it's basically trying to emulate splits/partitions for incremental ingestion? If yes, it would be better to support that with native splits or something similar – and then the actual input/output properties could just template in values from the split currently being executed. Does that make sense? And do you have any inputs or new discoveries for things to think about here?
2. Did you look into using Druid's INSERT or REPLACE SQL commands instead? I know it's less mature, but would be sweet if we could get away with only supporting the SQL interface.

[begelundmuller]

Connector names can be aliased (e.g. if you connect with two different service accounts for different buckets). So instead of checking InputConnector it should check InputHandle.Driver() instead.

[begelundmuller]

This is used, but not assigned anywhere

[begelundmuller]

Use e.connection.logger

[begelundmuller]

Could it use time.Time (it serializes/deserializes naturally in JSON)?

[begelundmuller]

So these are actually properties for an object store connector, not for the Druid connector. Ideally the name druid.ModelInputProperties would be reserved for input properties for Druid (i.e. models where Druid is the input and something else is the output – obviously we don't support that now, but could make sense in the future for e.g. models that export to S3).

Ideally the input properties for GCS would be defined in gcs.ModelInputProperties (or maybe drivers.ObjectStoreModelInputProperties if shared across multiple object store drivers), but I realize these are quite specific to this Druid driver. So maybe having it be an internal gcsModelInputProperties would make more sense.

[rohithreddykota]

Some high-level questions:

1. I didn't go deep on the time manipulation logic here, but it seems like it's basically trying to emulate splits/partitions for incremental ingestion? If yes, it would be better to support that with native splits or something similar – and then the actual input/output properties could just template in values from the split currently being executed. Does that make sense? And do you have any inputs or new discoveries for things to think about here?
2. Did you look into using Druid's INSERT or REPLACE SQL commands instead? I know it's less mature, but would be sweet if we could get away with only supporting the SQL interface.

1. You are right. It is kind of emulating splits for the incremental ingestion. Having native splits would definitely make things lot easier. From my understanding, splits should be able to serve all the options with a single query.
2. My initial idea was to add support for both sql based ingestion and specJSON based ingestion. The reason I want to add specJson based ingestion is that I can port the existing implementations into rill cloud. And another reasons are the limitations.