I wonder whether there are still memory leaks in the latest version ？

[wohaiaini]

I tested this lib(3.0.12) with nginx model ModSecurity-nginx on Linux platform by reloading nginx (nginx -s reload) again and again for quite a log time，and I found that the memory (RSS) still increases a little bit every several times. The more config rules, the bigger memory increases, and it will cause out of memory (OOM) reloading not more than a thousand times.
And I do read some issues like 2848 before, it seems the problem has been resolved, but I am not sure.

Here are some of my test results, and the docker env do cause OOM problem, and I don't know how that happens, could you please help me with that, many thanks.

the linux environment:

the docker environment（more Modsecurity rules）:

[S0obi]

Hello @wohaiaini,

First, please note that 3.0.12 is not the latest version, 3.0.14 has been recently released.
Memory leaks are indeed still an issue even with latest version and is already tracked by #2848.

The behavior you are mentioning is real and you are not the only one. The more modsecurity directives (like modsecurity_rules_file) you will have in your config, the bigger the leak.

The issue seems complex and multi-factorial, severals leaks have already been fixed in past versions but we need to continue this work together. Any help is appreciated.

[wohaiaini]

Okay, thanks for your reply. I will keep working on it.
If any progress is made, please let me know. Thank you very much.

[wohaiaini]

Hello @S0obi
I used some memory analysis tools (such as Valgrind, etc.), and the tools pointed out some potential issues. The most frequently occurring ones are shown in the following call stack, while there are others that I haven't listed. You may consider whether this is helpful in addressing the issue. Thank you.

The most frequently occurring ones are shown in the following call stack.
note: "src/seclang-parser.cc" may refer to "src/parser/seclang-parser.cc" line 1373

0	0x7aa30d	__gnu_cxx::new_allocator＜int＞::allocate(unsigned long, void const*) /usr/include/c++/8/ext/new_allocator.h:111
1	0x791a04	std::deque＜int, std::allocator＜int＞ ＞::push_back(int＆＆) /usr/include/c++/8/bits/stl_deque.h:1568
2	0x7675c9	yy::seclang_parser::parse() /root/modsecurity-v3.0.13/src/seclang-parser.cc:1373
3	0x751dcf	modsecurity::Parser::Driver::parse(std::__cxx11::basic_string＜char, std::char_traits＜char＞, std::allocator＜char＞ ＞ const＆, std::__cxx11::basic_string＜char, std::char_traits＜char＞, std::allocator＜char＞ ＞ const＆) /root/modsecurity-v3.0.13/src/parser/driver.cc:148
4	0x752105	modsecurity::Parser::Driver::parseFile(std::__cxx11::basic_string＜char, std::char_traits＜char＞, std::allocator＜char＞ ＞ const＆) /root/modsecurity-v3.0.13/src/parser/driver.cc:192 


   yybackup:
   1360     // Try to take a decision without lookahead.
   1361     yyn = yypact_[+yystack_[0].state];
   1362     if (yy_pact_value_is_default_ (yyn))
   1363       goto yydefault;
   1364 
   1365     // Read a lookahead token.
   1366     if (yyla.empty ())
   1367       {
   1368         YYCDEBUG << "Reading a token\n";
   1369 #if YY_EXCEPTIONS
   1370         try
   1371 #endif // YY_EXCEPTIONS
   1372           {
   1373             symbol_type yylookahead (yylex (driver));
   1374             yyla.move (yylookahead);
   1375           }
   1376 #if YY_EXCEPTIONS
   1377         catch (const syntax_error& yyexc)
   1378           {
   1379             YYCDEBUG << "Caught exception: " << yyexc.what() << '\n';
   1380             error (yyexc);
   1381             goto yyerrlab1;
   1382           }
   1383 #endif // YY_EXCEPTIONS
   1384       }

the second most frequently occurring ones are shown:

0	0x7748ac	yy::seclang_parser::parse() /root/modsecurity-v3.0.13/src/seclang-parser.yy:3069
1	0x751dcf	modsecurity::Parser::Driver::parse(std::__cxx11::basic_string＜char, std::char_traits＜char＞, std::allocator＜char＞ ＞ const＆, std::__cxx11::basic_string＜char, std::char_traits＜char＞, std::allocator＜char＞ ＞ const＆) /root/modsecurity-v3.0.13/src/parser/driver.cc:148
2	0x752105	modsecurity::Parser::Driver::parseFile(std::__cxx11::basic_string＜char, std::char_traits＜char＞, std::allocator＜char＞ ＞ const＆) /root/modsecurity-v3.0.13/src/parser/driver.cc:192
3	0x6fa82e	modsecurity::RulesSet::loadFromUri(char const*) /root/modsecurity-v3.0.13/src/rules_set.cc:53

0	0x752f6e	modsecurity::RunTimeString::appendText(std::__cxx11::basic_string＜char, std::char_traits＜char＞, std::allocator＜char＞ ＞ const＆) /root/modsecurity-v3.0.13/src/run_time_string.cc:34
1	0x7748ef	yy::seclang_parser::parse() /root/modsecurity-v3.0.13/src/seclang-parser.yy:3070
2	0x751dcf	modsecurity::Parser::Driver::parse(std::__cxx11::basic_string＜char, std::char_traits＜char＞, std::allocator＜char＞ ＞ const＆, std::__cxx11::basic_string＜char, std::char_traits＜char＞, std::allocator＜char＞ ＞ const＆) /root/modsecurity-v3.0.13/src/parser/driver.cc:148
3	0x752105	modsecurity::Parser::Driver::parseFile(std::__cxx11::basic_string＜char, std::char_traits＜char＞, std::allocator＜char＞ ＞ const＆) /root/modsecurity-v3.0.13/src/parser/driver.cc:192

0	0x752fc0	__gnu_cxx::new_allocator＜std::_List_node＜std::unique_ptr＜modsecurity::RunTimeElementHolder, std::default_delete＜modsecurity::RunTimeElementHolder＞ ＞ ＞ ＞::allocate(unsigned long, void const*) /usr/include/c++/8/ext/new_allocator.h:111
1	0x7748ef	yy::seclang_parser::parse() /root/modsecurity-v3.0.13/src/seclang-parser.yy:3070
2	0x751dcf	modsecurity::Parser::Driver::parse(std::__cxx11::basic_string＜char, std::char_traits＜char＞, std::allocator＜char＞ ＞ const＆, std::__cxx11::basic_string＜char, std::char_traits＜char＞, std::allocator＜char＞ ＞ const＆) /root/modsecurity-v3.0.13/src/parser/driver.cc:148
3	0x752105	modsecurity::Parser::Driver::parseFile(std::__cxx11::basic_string＜char, std::char_traits＜char＞, std::allocator＜char＞ ＞ const＆) /root/modsecurity-v3.0.13/src/parser/driver.cc:192

[S0obi]

@wohaiaini thanks for your report! Will be great if you can join the community on slack to work together on this issue. In parralel, can you document how you get this valgrind output so we can reproduce ? thanks in advance!

[wohaiaini]

Thanks for the invitation.
This tool is an in-house tool we've developed, similar to Valgrind, and I believe Valgrind could likely achieve comparable results. Additionally, since the security rules we use don't change frequently, we adopted an alternative strategy: by modifying the implementation of the ModSecurity-nginx module to ensure that only the changed rules are reloaded during a reload, temporarily circumventing the issue. The process works well for now, and I'll continue to monitor it closely.

[airween]

I also developed a tool but the aim is a bit different than check the memory usage (and possible memory leak(s)). ftwrunner's main goal is to run CRS checks against the "raw" library, but it's easy to use it with Valgrind. It's interesting that is does not show the leaks that you mentioned above. So I assume this is a connector issue, eg. it does not call msc_rules_cleanup when Nginx gets a HUP signal.