es_out: support Upstream Servers with configuration overriding #7608
Conversation
812020d to
ba3382a
Compare
|
Hi reviewers, Is it possible to approve only workflow for this pull request, so that automated checks and build can start? Thank you. |
|
@mabrarov sure |
|
Hi @PettitWesley, It looks like all failed checks are around run-macos-unit-tests jobs and caused by the following failed unit tests:
I feel like other pull requests have the same issues, i.e. it doesn't seem that the failed checks are caused by this pull request changes. Help of maintainers is appreciated. Thank you. |
ba3382a to
b7cd81b
Compare
|
Hi @PettitWesley, Is it possible to trigger automated workflow (build) for this pull request one more time? I found & fixed one issue and added tests for the new code since last build happened. Thank you. |
|
Hi dear reviewers, Is it possible to get this pull request reviewed / accepted sooner? Is there something pending / waiting from my side to start review? Thank you. |
b7cd81b to
b81d3f7
Compare
|
Hi @PettitWesley and @edsiper, It feels like you are code owners for Elasticsearch output plugin. Is there something pending / waiting from my side to start review of this pull request? This new feature was requested 4 years ago and I feel it is something which multiple users of Fluent Bit (not just my team) would like to have. Thank you. |
b81d3f7 to
f6431c2
Compare
73463c9 to
cb8d7b1
Compare
There was a problem hiding this comment.
We might want to use slightly different approach for achieving this.
This is because own_* flags are frequently appeared in this PR.
So, we wanted to wrap them with the following struct like:
/* A wrapper for flb_sds_t that tracks memory ownership */
typedef struct {
flb_sds_t sds; /* The actual string data */
int owned; /* A flag indicating if this struct owns the memory */
} flb_es_sds_t;Then, the wrapped members could be:
/* In es.h, struct flb_elasticsearch_config */
struct flb_elasticsearch_config {
/* ... other properties ... */
flb_es_sds_t index;
flb_es_sds_t type;
flb_es_sds_t http_user;
flb_es_sds_t logstash_prefix;
flb_es_sds_t time_key;
/* ... etc ... */
};For just initialization, it could be:
static inline void es_sds_init(flb_es_sds_t *dest, flb_es_sds_t *src)
{
dest->sds = src->sds;
dest->owned = FLB_FALSE; /* The new struct does not own the shared memory */
}For setting up a value, it could be:
static inline int es_sds_set(flb_es_sds_t *wrapper, const char *value)
{
/* If we previously owned a different string, free it first */
if (wrapper->owned && wrapper->sds) {
flb_sds_destroy(wrapper->sds);
}
wrapper->sds = flb_sds_create(value);
if (!wrapper->sds) {
return -1;
}
wrapper->owned = FLB_TRUE;
return 0;
}The teardown operation of these values could be:
static inline void es_sds_destroy(flb_es_sds_t *wrapper)
{
if (wrapper->owned && wrapper->sds) {
flb_sds_destroy(wrapper->sds);
}
wrapper->sds = NULL;
wrapper->owned = FLB_FALSE;
}Then, just teardown operation could be:
es_sds_destroy(&ec->index);
es_sds_destroy(&ec->type);
/* ... etc ... */These changes could be cleaner way to handle shallow copy.
We would love to use this type of wrapper to encapsulate for these type of complex operations.
|
Hi @cosmo0920, Regarding:
I like this approach, but there might be not just Thank you. |
Hi, |
|
Hi @cosmo0920, Here is one more thing I found. Some of configuration options - like fluent-bit/plugins/out_es/es.c Line 1077 in 42e29d6 If I use custom type for such configuration options (fields of struct flb_es_char_ptr {
char *ptr;
int owned;
};
struct flb_elasticsearch_config {
/* Elasticsearch index (database) and type (table) */
struct flb_es_char_ptr index;
...
};then it will complicate usage of static struct flb_config_map config_map[] = {
{
FLB_CONFIG_MAP_STR, FLB_ES_CONFIG_PROPERTY_INDEX, FLB_ES_DEFAULT_INDEX,
0, FLB_TRUE, offsetof(struct flb_elasticsearch_config, index) + offsetof(struct flb_es_char_ptr, ptr),
"Set an index name"
},
...Anyway, I need to try it first. Just some thoughts. Thank you. |
…wn to parser of Upstream node configuration section are implemented, e.g. "host" and "port" Signed-off-by: Marat Abrarov <[email protected]>
…o the test callback based on configuration of Fluent Bit and based on configuration of plugin Signed-off-by: Marat Abrarov <[email protected]>
…llback contract Signed-off-by: Marat Abrarov <[email protected]>
…with Upstream node configuration For Elastic cloud authentication these parameters are always taken from plugin configuration and never from Upstream node configuration: cloud_id. For AWS authentication these parameters are always taken from plugin configuration and never from Upstream node configuration: http_proxy, no_proxy, tls*. Signed-off-by: Marat Abrarov <[email protected]>
Signed-off-by: Marat Abrarov <[email protected]>
…o the test callback based on configuration of Fluent Bit and based on configuration of plugin Signed-off-by: Marat Abrarov <[email protected]>
cb8d7b1 to
717d6b3
Compare
WalkthroughThis update refactors and modularizes the Elasticsearch output plugin in Fluent Bit, introducing new configuration structures and helper files to support high-availability (HA) mode and upstream configuration overrides. It adds new test cases for upstream-specific properties, updates function signatures and callbacks for test mode, and introduces several new headers and helper functions for configuration parsing and AWS/cloud integration. Changes
Sequence Diagram(s)sequenceDiagram
participant TestRunner
participant OutputInstance
participant FlushCtxCallback
participant FormatterCallback
TestRunner->>OutputInstance: Enable test mode with flush_ctx_callback
TestRunner->>OutputInstance: Push test data
OutputInstance->>FlushCtxCallback: Get flush context (if callback set)
FlushCtxCallback-->>OutputInstance: Return flush context
OutputInstance->>FormatterCallback: Run formatter with flush context
FormatterCallback-->>OutputInstance: Return result
OutputInstance-->>TestRunner: Test result
Estimated code review effort🎯 5 (Critical) | ⏱️ ~75 minutes Poem
Note ⚡️ Unit Test Generation is now available in beta!Learn more here, or try it out under "Finishing Touches" below. ✨ Finishing Touches
🧪 Generate unit tests
🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
There was a problem hiding this comment.
Actionable comments posted: 3
♻️ Duplicate comments (4)
plugins/out_es/es.h (1)
48-161: Ownership flags pattern needs architectural consideration.The extensive use of
own_xxxflags throughout the struct deviates from Fluent Bit's typical patterns. While functional for tracking resource ownership between plugin and upstream configurations, this approach adds complexity that may impact maintainability.As discussed in previous reviews, consider whether this ownership tracking could be encapsulated differently or if the architecture needs adaptation to properly support this use case.
plugins/out_es/es_conf_parse.c (2)
102-109: Consider using thread-safe string parsing alternatives.The use of
strtokis not thread-safe and has been flagged as a dangerous function. Consider usingflb_utils_splitor other thread-safe alternatives for better safety.
124-130: Use safer string operations to prevent buffer overflows.Using
strcpyandstrcatwithout bounds checking can lead to buffer overflows. Consider usingsnprintfor Fluent Bit'sflb_sdsAPIs for safer string operations.- strcpy(cloud_host_buf, host); - strcat(cloud_host_buf, "."); - strcat(cloud_host_buf, region); - if (port != NULL) { - strcat(cloud_host_buf, ":"); - strcat(cloud_host_buf, port); - } + int ret; + if (port != NULL) { + ret = snprintf(cloud_host_buf, sizeof(cloud_host_buf), "%s.%s:%s", + host, region, port); + } else { + ret = snprintf(cloud_host_buf, sizeof(cloud_host_buf), "%s.%s", + host, region); + } + if (ret < 0 || ret >= sizeof(cloud_host_buf)) { + return NULL; + }plugins/out_es/es_conf.c (1)
302-327: Ownership tracking pattern needs architectural reviewAs noted in previous reviews, this ownership tracking pattern using
own_xxxflags deviates from Fluent Bit's typical coding style. While the implementation appears correct for preventing double-frees in a shared configuration scenario, this approach introduces complexity that may be difficult to maintain.Consider:
- Documenting why this pattern is necessary for the Elasticsearch plugin specifically
- Exploring alternative approaches such as reference counting or configuration cloning
- Adding comprehensive comments explaining the ownership model
🧹 Nitpick comments (3)
include/fluent-bit/flb_lib.h (1)
74-80: Consider function signature readability.The new function signature is quite complex with many parameters. While this follows existing patterns in the codebase for similar callback registration functions, consider if some parameters could be grouped into a struct for better maintainability.
However, this is consistent with the existing API style in this file and likely maintains backward compatibility.
plugins/out_es/es.c (1)
853-867: Consider simplifying upstream connection logicThe current if-else structure could be simplified by extracting the upstream selection logic.
- /* Get upstream connection */ - if (ctx->ha_mode == FLB_TRUE) { - u_conn = flb_upstream_conn_get(node->u); - if (!u_conn) { - flb_plg_error(ctx->ins, "no upstream connections available for %s node", - node->name); - FLB_OUTPUT_RETURN(FLB_RETRY); - } - } - else { - u_conn = flb_upstream_conn_get(ctx->u); - if (!u_conn) { - flb_plg_error(ctx->ins, "no upstream connections available"); - FLB_OUTPUT_RETURN(FLB_RETRY); - } - } + /* Get upstream connection */ + struct flb_upstream *upstream = (ctx->ha_mode == FLB_TRUE) ? node->u : ctx->u; + u_conn = flb_upstream_conn_get(upstream); + if (!u_conn) { + if (ctx->ha_mode == FLB_TRUE) { + flb_plg_error(ctx->ins, "no upstream connections available for %s node", + node->name); + } else { + flb_plg_error(ctx->ins, "no upstream connections available"); + } + FLB_OUTPUT_RETURN(FLB_RETRY); + }tests/runtime/out_elasticsearch.c (1)
1065-1067: Add validation for upstream configuration loadingThe test creates an upstream configuration file but doesn't verify if it was successfully loaded and parsed. Consider adding assertions to validate the upstream configuration was applied correctly.
// After setting upstream, verify it was loaded TEST_CHECK(ctx != NULL); // Could also add checks to verify the upstream nodes were created
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (13)
include/fluent-bit/flb_lib.h(2 hunks)include/fluent-bit/flb_output.h(1 hunks)plugins/out_es/CMakeLists.txt(1 hunks)plugins/out_es/es.c(31 hunks)plugins/out_es/es.h(5 hunks)plugins/out_es/es_conf.c(1 hunks)plugins/out_es/es_conf.h(1 hunks)plugins/out_es/es_conf_parse.c(1 hunks)plugins/out_es/es_conf_parse.h(1 hunks)plugins/out_es/es_conf_prop.h(1 hunks)src/flb_engine_dispatch.c(2 hunks)src/flb_lib.c(2 hunks)tests/runtime/out_elasticsearch.c(17 hunks)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (70)
- GitHub Check: PR - container builds / Windows container images (2025)
- GitHub Check: PR - container builds / Windows container images (2022)
- GitHub Check: PR - packages build Windows / call-build-windows-package (Windows 64bit, x64, x64-windows-static, 3.31.6)
- GitHub Check: PR - packages build Windows / call-build-windows-package (Windows 64bit (Arm64), amd64_arm64, -DCMAKE_SYSTEM_NAME=Windows -DCMA...
- GitHub Check: PR - packages build Windows / call-build-windows-package (Windows 32bit, x86, x86-windows-static, 3.31.6)
- GitHub Check: PR - packages build Linux / ubuntu/24.04 package build and stage to S3
- GitHub Check: PR - packages build Linux / raspbian/bookworm package build and stage to S3
- GitHub Check: PR - packages build Linux / ubuntu/24.04.arm64v8 package build and stage to S3
- GitHub Check: PR - packages build Linux / debian/bullseye.arm64v8 package build and stage to S3
- GitHub Check: PR - packages build Linux / ubuntu/22.04 package build and stage to S3
- GitHub Check: PR - packages build Linux / rockylinux/9.arm64v8 package build and stage to S3
- GitHub Check: PR - packages build Linux / debian/bullseye package build and stage to S3
- GitHub Check: PR - packages build Linux / debian/bookworm package build and stage to S3
- GitHub Check: PR - packages build Linux / ubuntu/22.04.arm64v8 package build and stage to S3
- GitHub Check: PR - packages build Linux / debian/buster package build and stage to S3
- GitHub Check: PR - packages build Linux / debian/buster.arm64v8 package build and stage to S3
- GitHub Check: PR - packages build Linux / almalinux/8.arm64v8 package build and stage to S3
- GitHub Check: PR - packages build Linux / almalinux/8 package build and stage to S3
- GitHub Check: PR - packages build Linux / almalinux/9.arm64v8 package build and stage to S3
- GitHub Check: PR - packages build Linux / debian/bookworm.arm64v8 package build and stage to S3
- GitHub Check: PR - packages build Linux / rockylinux/9 package build and stage to S3
- GitHub Check: PR - packages build Linux / rockylinux/8 package build and stage to S3
- GitHub Check: PR - packages build Linux / centos/7 package build and stage to S3
- GitHub Check: PR - packages build Linux / almalinux/9 package build and stage to S3
- GitHub Check: PR - packages build Linux / rockylinux/8.arm64v8 package build and stage to S3
- GitHub Check: PR - packages build Linux / centos/7.arm64v8 package build and stage to S3
- GitHub Check: PR - packages build Linux / centos/8 package build and stage to S3
- GitHub Check: PR - packages build Linux / amazonlinux/2.arm64v8 package build and stage to S3
- GitHub Check: PR - packages build Linux / amazonlinux/2023 package build and stage to S3
- GitHub Check: PR - packages build Linux / amazonlinux/2023.arm64v8 package build and stage to S3
- GitHub Check: PR - packages build Linux / centos/9 package build and stage to S3
- GitHub Check: PR - packages build Linux / centos/9.arm64v8 package build and stage to S3
- GitHub Check: PR - packages build Linux / centos/8.arm64v8 package build and stage to S3
- GitHub Check: PR - packages build Linux / amazonlinux/2 package build and stage to S3
- GitHub Check: PR - container builds / arm64/debug container image build
- GitHub Check: PR - container builds / arm/v7/production container image build
- GitHub Check: PR - container builds / arm/v7/debug container image build
- GitHub Check: PR - container builds / amd64/debug container image build
- GitHub Check: PR - container builds / arm64/production container image build
- GitHub Check: PR - container builds / amd64/production container image build
- GitHub Check: PR - packages build MacOS / call-build-macos-package (Intel macOS runner, macos-14-large, 3.31.6)
- GitHub Check: PR - packages build MacOS / call-build-macos-package (Apple Silicon macOS runner, macos-14, 3.31.6)
- GitHub Check: pr-windows-build / call-build-windows-package (Windows 64bit (Arm64), amd64_arm64, -DCMAKE_SYSTEM_NAME=Windows -DCMA...
- GitHub Check: pr-windows-build / call-build-windows-package (Windows 32bit, x86, x86-windows-static, 3.31.6)
- GitHub Check: pr-windows-build / call-build-windows-package (Windows 64bit, x64, x64-windows-static, 3.31.6)
- GitHub Check: pr-compile-system-libs (-DFLB_PREFER_SYSTEM_LIBS=On, 3.31.6, clang, clang++, ubuntu-24.04, clang-14)
- GitHub Check: pr-compile-system-libs (-DFLB_PREFER_SYSTEM_LIBS=On, 3.31.6, gcc, g++, ubuntu-22.04, clang-12)
- GitHub Check: pr-compile-system-libs (-DFLB_PREFER_SYSTEM_LIBS=On, 3.31.6, clang, clang++, ubuntu-22.04, clang-12)
- GitHub Check: pr-compile-system-libs (-DFLB_PREFER_SYSTEM_LIBS=On, 3.31.6, gcc, g++, ubuntu-24.04, clang-14)
- GitHub Check: pr-compile-centos-7
- GitHub Check: run-ubuntu-unit-tests (-DFLB_SIMD=Off, 3.31.6, gcc, g++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_SIMD=On, 3.31.6, clang, clang++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_SIMD=Off, 3.31.6, clang, clang++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_SANITIZE_MEMORY=On, 3.31.6, clang, clang++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_SANITIZE_THREAD=On, 3.31.6, gcc, g++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_SANITIZE_MEMORY=On, 3.31.6, gcc, g++)
- GitHub Check: run-ubuntu-unit-tests (-DSANITIZE_UNDEFINED=On, 3.31.6, gcc, g++)
- GitHub Check: run-ubuntu-unit-tests (-DSANITIZE_ADDRESS=On, 3.31.6, gcc, g++)
- GitHub Check: run-ubuntu-unit-tests (-DSANITIZE_UNDEFINED=On, 3.31.6, clang, clang++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_SIMD=On, 3.31.6, gcc, g++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_COVERAGE=On, 3.31.6, gcc, g++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_SANITIZE_THREAD=On, 3.31.6, clang, clang++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_SMALL=On, 3.31.6, clang, clang++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_SMALL=On, 3.31.6, gcc, g++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_JEMALLOC=Off, 3.31.6, clang, clang++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_JEMALLOC=On, 3.31.6, clang, clang++)
- GitHub Check: run-ubuntu-unit-tests (-DSANITIZE_ADDRESS=On, 3.31.6, clang, clang++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_JEMALLOC=Off, 3.31.6, gcc, g++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_JEMALLOC=On, 3.31.6, gcc, g++)
- GitHub Check: PR - fuzzing test
🔇 Additional comments (15)
plugins/out_es/es_conf_prop.h (1)
1-65: LGTM! Well-structured configuration property definitions.The header file follows good practices with proper include guards, consistent naming conventions, and appropriate conditional compilation for AWS features.
plugins/out_es/CMakeLists.txt (1)
3-3: LGTM! Correct addition of new source file.The new
es_conf_parse.csource file is properly added to the build configuration in alphabetical order.include/fluent-bit/flb_lib.h (2)
25-25: LGTM! Required include for new API.The addition of
flb_input.his necessary for thestruct flb_input_instancetype used in the new flush context callback function.
73-73: LGTM! Parameter rename improves clarity.Renaming
test_ctxtoflush_ctxbetter reflects the parameter's purpose in the context of flush operations.plugins/out_es/es_conf.h (4)
23-23: LGTM! Appropriate include for upstream functionality.The replacement of general includes with
flb_upstream_node.haligns with the focus on upstream server support.
27-30: LGTM! Well-defined write operation constants.The write operation constants are clearly defined and follow consistent naming conventions. These align with standard Elasticsearch operations.
35-35: LGTM! Appropriate return type change.Changing
flb_es_conf_destroyto void suggests improved error handling where cleanup operations are designed to always succeed or handle errors internally.
37-38: LGTM! New function supports upstream configuration.The new
flb_es_upstream_conffunction provides a clean interface for retrieving upstream-specific configuration, which is essential for the upstream servers feature.include/fluent-bit/flb_output.h (2)
153-154: LGTM! Clarified comment improves understanding.The updated comment clearly explains the relationship between
flush_ctxand the new callback mechanism.
156-169: LGTM! Well-documented callback interface.The new
flush_ctx_callbackfunction pointer is properly documented with clear parameter descriptions. The signature is consistent with the API declared inflb_lib.h.src/flb_engine_dispatch.c (1)
101-142: Changes to test formatter function look good.The refactoring to retrieve
flush_ctxlocally and support an optional callback provides better flexibility for test configurations without breaking existing functionality.src/flb_lib.c (1)
579-644: API changes for test mode configuration are well-implemented.The parameter rename to
flush_ctximproves clarity, and the newflb_output_set_test_flush_ctx_callbackfunction follows established patterns with proper error handling.plugins/out_es/es_conf_parse.h (1)
1-54: Header file structure and declarations are appropriate.The new header properly declares configuration parsing functions with clear documentation and appropriate conditional compilation for AWS features.
plugins/out_es/es.h (1)
28-28: Good catch on fixing the default port.The correction from 92000 to 9200 fixes what was clearly a typo in the default Elasticsearch port.
plugins/out_es/es_conf_parse.c (1)
186-337: AWS authentication setup is comprehensive.The AWS provider initialization with proper TLS context creation, STS support, and error handling is well-implemented. Good use of ownership flags to track resource lifecycle.
| ec->logstash_prefix = flb_sds_create(tmp); | ||
| if (ec->logstash_prefix == NULL) { | ||
| return -1; | ||
| } | ||
| ec->own_logstash_prefix = FLB_TRUE; | ||
| } | ||
| tmp = flb_upstream_node_get_property(FLB_ES_CONFIG_PROPERTY_LOGSTASH_PREFIX_SEPARATOR, | ||
| node); | ||
| if (tmp) { | ||
| ec->logstash_prefix_separator = flb_sds_create(tmp); | ||
| if (ec->logstash_prefix_separator == NULL) { | ||
| return -1; | ||
| } | ||
| ec->own_logstash_prefix_separator = FLB_TRUE; | ||
| } | ||
| tmp = flb_upstream_node_get_property(FLB_ES_CONFIG_PROPERTY_LOGSTASH_DATEFORMAT, | ||
| node); | ||
| if (tmp) { | ||
| ec->logstash_dateformat = flb_sds_create(tmp); | ||
| if (ec->logstash_dateformat == NULL) { | ||
| return -1; | ||
| } | ||
| ec->own_logstash_dateformat = FLB_TRUE; | ||
| } | ||
| tmp = flb_upstream_node_get_property(FLB_ES_CONFIG_PROPERTY_TIME_KEY, node); | ||
| if (tmp) { | ||
| ec->time_key = flb_sds_create(tmp); | ||
| if (ec->time_key == NULL) { | ||
| return -1; | ||
| } | ||
| ec->own_time_key = FLB_TRUE; | ||
| } | ||
| tmp = flb_upstream_node_get_property(FLB_ES_CONFIG_PROPERTY_TIME_KEY_FORMAT, node); | ||
| if (tmp) { | ||
| ec->time_key_format = flb_sds_create(tmp); | ||
| if (ec->time_key_format == NULL) { | ||
| return -1; | ||
| } | ||
| ec->own_time_key_format = FLB_TRUE; | ||
| } | ||
| tmp = flb_upstream_node_get_property(FLB_ES_CONFIG_PROPERTY_TIME_KEY_NANOS, node); | ||
| if (tmp) { | ||
| ec->time_key_nanos = flb_utils_bool(tmp); | ||
| } | ||
| tmp = flb_upstream_node_get_property(FLB_ES_CONFIG_PROPERTY_INCLUDE_TAG_KEY, node); | ||
| if (tmp) { | ||
| ec->include_tag_key = flb_utils_bool(tmp); | ||
| } | ||
| tmp = flb_upstream_node_get_property(FLB_ES_CONFIG_PROPERTY_TAG_KEY, node); | ||
| if (tmp) { | ||
| ec->tag_key = flb_sds_create(tmp); | ||
| if (ec->tag_key == NULL) { | ||
| return -1; | ||
| } | ||
| ec->own_tag_key = FLB_TRUE; |
There was a problem hiding this comment.
🛠️ Refactor suggestion
Consider cleanup on partial allocation failures
When creating multiple allocated strings, if one allocation fails (e.g., at line 444), previously allocated strings in the same ec structure (like logstash_prefix at line 409) won't be freed before returning -1.
Consider adding a cleanup path or ensuring the caller properly handles partial initialization:
// At the error return points, ensure cleanup of already allocated resources
if (ec->tag_key == NULL) {
// Free any previously allocated strings in this function
if (ec->own_logstash_prefix == FLB_TRUE) {
flb_sds_destroy(ec->logstash_prefix);
ec->own_logstash_prefix = FLB_FALSE;
}
// ... similar cleanup for other allocated strings
return -1;
}🤖 Prompt for AI Agents
In plugins/out_es/es_conf.c between lines 409 and 463, multiple string
allocations are performed and if any allocation fails, the function returns -1
without freeing previously allocated strings, causing potential memory leaks. To
fix this, add a cleanup section before each error return that checks which
strings were successfully allocated (using the corresponding own_* flags) and
frees them properly, resetting the flags to false. This ensures no partial
allocations remain unfreed on failure.
| struct flb_elasticsearch_config *flb_elasticsearch_target( | ||
| struct flb_elasticsearch *ctx, struct flb_upstream_node **node) | ||
| { | ||
| struct flb_elasticsearch_config *ec; | ||
| struct flb_upstream_node *target_node; | ||
|
|
||
| if (ctx->ha_mode == FLB_FALSE) { | ||
| ec = flb_es_upstream_conf(ctx, NULL); | ||
| *node = NULL; | ||
| return ec; | ||
| } | ||
|
|
||
| target_node = flb_upstream_ha_node_get(ctx->ha); | ||
| if (!target_node) { | ||
| *node = NULL; | ||
| return NULL; | ||
| } | ||
|
|
||
| ec = flb_es_upstream_conf(ctx, target_node); | ||
| *node = target_node; | ||
|
|
||
| return ec; | ||
| } |
There was a problem hiding this comment.
Add null check for ctx parameter
The function should validate the ctx parameter before dereferencing it to avoid potential null pointer dereference.
struct flb_elasticsearch_config *flb_elasticsearch_target(
struct flb_elasticsearch *ctx, struct flb_upstream_node **node)
{
struct flb_elasticsearch_config *ec;
struct flb_upstream_node *target_node;
+ if (!ctx || !node) {
+ return NULL;
+ }
+
if (ctx->ha_mode == FLB_FALSE) {
ec = flb_es_upstream_conf(ctx, NULL);
*node = NULL;
return ec;
}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| struct flb_elasticsearch_config *flb_elasticsearch_target( | |
| struct flb_elasticsearch *ctx, struct flb_upstream_node **node) | |
| { | |
| struct flb_elasticsearch_config *ec; | |
| struct flb_upstream_node *target_node; | |
| if (ctx->ha_mode == FLB_FALSE) { | |
| ec = flb_es_upstream_conf(ctx, NULL); | |
| *node = NULL; | |
| return ec; | |
| } | |
| target_node = flb_upstream_ha_node_get(ctx->ha); | |
| if (!target_node) { | |
| *node = NULL; | |
| return NULL; | |
| } | |
| ec = flb_es_upstream_conf(ctx, target_node); | |
| *node = target_node; | |
| return ec; | |
| } | |
| struct flb_elasticsearch_config *flb_elasticsearch_target( | |
| struct flb_elasticsearch *ctx, struct flb_upstream_node **node) | |
| { | |
| struct flb_elasticsearch_config *ec; | |
| struct flb_upstream_node *target_node; | |
| if (!ctx || !node) { | |
| return NULL; | |
| } | |
| if (ctx->ha_mode == FLB_FALSE) { | |
| ec = flb_es_upstream_conf(ctx, NULL); | |
| *node = NULL; | |
| return ec; | |
| } | |
| target_node = flb_upstream_ha_node_get(ctx->ha); | |
| if (!target_node) { | |
| *node = NULL; | |
| return NULL; | |
| } | |
| ec = flb_es_upstream_conf(ctx, target_node); | |
| *node = target_node; | |
| return ec; | |
| } |
🤖 Prompt for AI Agents
In plugins/out_es/es.c around lines 645 to 667, the function
flb_elasticsearch_target dereferences the ctx parameter without checking if it
is NULL, which can cause a null pointer dereference. Add a null check at the
start of the function to verify if ctx is NULL, and if so, set *node to NULL and
return NULL immediately to safely handle invalid input.
| int arg_idx; | ||
| va_list args; | ||
|
|
||
| upstream_conf_filename = tmpnam(NULL); |
There was a problem hiding this comment.
Replace deprecated tmpnam() with safer alternative
The tmpnam() function is deprecated and has known security vulnerabilities (race conditions, predictable names). Use mkstemp() or platform-specific secure alternatives instead.
- upstream_conf_filename = tmpnam(NULL);
- if (upstream_conf_filename == NULL) {
- return NULL;
- }
-
- upstream_conf_file = fopen(upstream_conf_filename, "w");
+ char template[] = "/tmp/flb_es_test_XXXXXX";
+ int fd = mkstemp(template);
+ if (fd == -1) {
+ return NULL;
+ }
+
+ upstream_conf_filename = strdup(template);
+ if (upstream_conf_filename == NULL) {
+ close(fd);
+ unlink(template);
+ return NULL;
+ }
+
+ upstream_conf_file = fdopen(fd, "w");
if (upstream_conf_file == NULL) {
+ close(fd);
+ unlink(template);
+ free(upstream_conf_filename);
return NULL;
}🤖 Prompt for AI Agents
In tests/runtime/out_elasticsearch.c at line 32, replace the deprecated and
insecure tmpnam() call with mkstemp() to safely create a unique temporary
filename. Use mkstemp() to generate a secure temporary file, handle the returned
file descriptor properly, and store the filename for further use, ensuring no
race conditions or predictable names.
Implementation of Upstream feature for the Elasticsearch output plugin.
This pull request is based on pull request #1560 and Forward output plugin.
It was tested in a local setup with:
Fluent Bit without Upstream feature connected to a single node of Elasticsearch cluster consisting of 3 master-eligible/data and 1 coordinating nodes.
Refer to elastic-cluster directory of mabrarov/elastic-stack repository for Docker Compose project used to create target Elasticsearch cluster and Kibana.
fluent-bit.conf Fluent Bit configuration file used for the test - refer to fluent-bit-es/fluent-bit.conf and (same in YAML format) fluent-bit-es/fluent-bit.yaml in mabrarov/elastic-stack repository.
Debug log is available at flb_es.log.
Fluent Bit with Upstream feature connected to all Elasticsearch data nodes of Elasticsearch cluster consisting of 3 master-eligible/data and 1 coordinating nodes.
Refer to elastic-cluster directory of mabrarov/elastic-stack repository for Docker Compose project used to create target Elasticsearch cluster and Kibana.
fluent-bit.conf Fluent Bit configuration file used for the test - refer to fluent-bit-es-cluster/fluent-bit.conf and (same in YAML format) fluent-bit-es-cluster/fluent-bit.yaml in mabrarov/elastic-stack repository.
Debug log is available at flb_es_upstream.log.
Testing
TEST_PRESET=valgrind SKIP_TESTS='flb-rt-out_td flb-it-network' ./run_code_analysis.shok-package-testlabel to test for all targets (requires maintainer to do).Documentation
Backporting
Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.