-
Notifications
You must be signed in to change notification settings - Fork 111
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
druid ingest executor #5234
base: main
Are you sure you want to change the base?
druid ingest executor #5234
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Some high-level questions:
- I didn't go deep on the time manipulation logic here, but it seems like it's basically trying to emulate splits/partitions for incremental ingestion? If yes, it would be better to support that with native splits or something similar – and then the actual input/output properties could just template in values from the split currently being executed. Does that make sense? And do you have any inputs or new discoveries for things to think about here?
- Did you look into using Druid's
INSERT
orREPLACE
SQL commands instead? I know it's less mature, but would be sweet if we could get away with only supporting the SQL interface.
@@ -256,12 +257,19 @@ func (c *connection) AsObjectStore() (drivers.ObjectStore, bool) { | |||
|
|||
// AsModelExecutor implements drivers.Handle. | |||
func (c *connection) AsModelExecutor(instanceID string, opts *drivers.ModelExecutorOptions) (drivers.ModelExecutor, bool) { | |||
if opts.OutputHandle == c && opts.InputConnector == "gcs" { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Connector names can be aliased (e.g. if you connect with two different service accounts for different buckets). So instead of checking InputConnector
it should check InputHandle.Driver()
instead.
db *sqlx.DB | ||
config *configProperties | ||
logger *zap.Logger | ||
instanceID string |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is used, but not assigned anywhere
fmt.Println("==>inputProperties", inputProperties) | ||
fmt.Println("==>outputProperties", outputProperties) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Use e.connection.logger
PreviousExecutionTime string `mapstructure:"previous_execution_time"` | ||
PreviousIntervalEndTime string `mapstructure:"previous_interval_end_time"` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could it use time.Time
(it serializes/deserializes naturally in JSON)?
type ModelInputProperties struct { | ||
Path string `mapstructure:"path"` | ||
Pattern string `mapstructure:"pattern"` | ||
Granularity string `mapstructure:"gran"` | ||
Format string `mapstructure:"format"` | ||
FilePattern string `mapstructure:"file_pattern"` | ||
RetriesPeriod string `mapstructure:"retry_period"` | ||
MaxRetries int `mapstructure:"max_retries"` | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So these are actually properties for an object store connector, not for the Druid connector. Ideally the name druid.ModelInputProperties
would be reserved for input properties for Druid (i.e. models where Druid is the input and something else is the output – obviously we don't support that now, but could make sense in the future for e.g. models that export to S3).
Ideally the input properties for GCS would be defined in gcs.ModelInputProperties
(or maybe drivers.ObjectStoreModelInputProperties
if shared across multiple object store drivers), but I realize these are quite specific to this Druid driver. So maybe having it be an internal gcsModelInputProperties
would make more sense.
|
Summary
This PR introduces a druidIndexExecutor which is responsible for executing Druid index tasks using configurations from YAML files. The executor handles the ingestion of data from an object store into Druid, calculating necessary intervals, generating dynamic index JSON, and performing the ingestion process based on the model input and output configurations.
Example: