Openvasmd authentication error after starting big task

Indeed :slight_smile:
Let me know if it works

Giovanni

Hi Giovanni,

Just to keep you updated;
I have recompiled the source of the latest stable version available (GVM-9) and I confirm the problem is still there. It seems it’s a bug introduced in GVM-9, as this wasn’t happening prior upgrading to this version.

There seems to be a fix in the master branch (https://github.com/greenbone/gvmd/pull/338)

But unfortunately, it doesn’t seem to have been backported.

As written in the topic Sqlite error / manager daemon authentication error where the posts of SQLite error and tasks stuck where split off the linked PR is unrelated to your seen issue here. This PR is only fixing an issue where tasks where not started at all when using schedules.

Referring to SQLite:

I think the problem is that the SQLite answer to OpenVas the code: SQLITE_BUSY and after 10 retries Openvas close the connection and return the error.
In sql_sqlite3.c I have noticed these possible problems:

  1. The timeout is hardcoded

#define BUSY_TIMEOUT 1000

Openvas has a config file, maybe it could be a good choice to put this configuration in this file

  1. Line 441 sql_sqlite3.c in the function sql_exec_internal

int sql_exec_internal (int retry, sql_stmt_t *stmt)
[…]
while (1)
{
int ret;
ret = sqlite3_step (stmt->stmt);
if (ret == SQLITE_BUSY)
{
if (retry)
{
if ((retries > 10) && (OPENVAS_SQLITE_SLEEP_MAX > 0))
openvas_usleep (MIN ((retries - 10) * 10000,
OPENVAS_SQLITE_SLEEP_MAX));
retries++;
continue;
}
if (retries++ < 10)
continue;
return -2;
}
if (retry == 0)
sqlite3_busy_timeout (task_db, BUSY_TIMEOUT);
if (ret == SQLITE_DONE)
return 0;
if (ret == SQLITE_ROW)
return 1;
g_warning (“%s: sqlite3_step failed: %s\n”,
FUNCTION,
sqlite3_errmsg (task_db));
return -1;
}

Tha variable “OPENVAS_SQLITE_SLEEP_MAX” is never set or, if it set, the default is 0.
checking in my code, I have this code in CMakeLists.txt (line 167 and 168):

if (NOT DEFINED OPENVAS_SQLITE_SLEEP_MAX)
set (OPENVAS_SQLITE_SLEEP_MAX 0)

This means that Openvas doesn’t retry to connect to DB, but only abort.
I have checked in the Github code, but in CMakeLists.txt I haven’t seen the OPENVAS_SQLITE_SLEEP_MAX 0

If you modify the code in sql_sqlite3.c line 435:

if ((retries > 10) && (OPENVAS_SQLITE_SLEEP_MAX > 0))

in

if ( ((retries > 10) && (OPENVAS_SQLITE_SLEEP_MAX > 0))
OR (ret == SQLITE_BUSY) )

The workaround should work (but it’s not the correct way to solve it)

1 Like

I’ll try to check if the workaround work also for postgres…

:Thanks cfi for clarifying. Too bad :frowning:

Thanks Mogimbo for your investigation, but it’s a bit cryptic for me. It seems difficult to understand that sqlite become busy with so few tasks / requests. Especially since the problem wasn’t happening with previous versions of OpenVAS… If on top of this Postgresql suffer the same issue (which is a robust db management system), then the problem has to be on OpenVAS side, not on the db side.

Or am I misunderstanding something ?

Thanks

Hi Tatooin,
I have done this kind of test:
I have created some PHP pages to query directly the PostgresSQL and (Slowly) it answer.
I also configured the PostgresSQL with some “best practice” found in internet.
Using my PHP pages, Openvas answer, so I think it’s not a problem of PostgresQL; maybe is how the DB of Openvas is build and how many data has.
remember that even if tasks are few, you have a lot of information regarding:

  1. CVE
  2. CPE
  3. NVT
  4. Etcetera
  5. Etcetera

I think that the error we received is based on a timeout on Openvas Manager.

I agree. Having said that, is there any way to increase openvasmd timeout ? This is becoming really problematic in my day to day job. The only option I find is “–schedule-timeout” and possibly “–client-watch-interval=” but I don’t think it’s related.

Thanks

I think you have to change the source code in sql__sqlite3.c as I explained before.
If you need more help just ask :smiley:

Ps. My workaround is tested only for SQLite; I have to check for Postgres

I’ve modified the sql_sqlite3.c as you said, but compiling the file fails with the following error:

[ 60%] Building C object src/CMakeFiles/manage.dir/sql_sqlite3.c.o
/root/gvm-9/gvmd-openvas-manager-7.0/src/sql_sqlite3.c: In function ‘sql_exec_internal’:
/root/gvm-9/gvmd-openvas-manager-7.0/src/sql_sqlite3.c:449:59: error: expected ‘)’ before ‘OR’
if ( ((retries > 10) && (OPENVAS_SQLITE_SLEEP_MAX > 0)) OR (ret == SQLITE_BUSY) )
^~
src/CMakeFiles/manage.dir/build.make:422: recipe for target ‘src/CMakeFiles/manage.dir/sql_sqlite3.c.o’ failed
make[2]: *** [src/CMakeFiles/manage.dir/sql_sqlite3.c.o] Error 1
CMakeFiles/Makefile2:244: recipe for target ‘src/CMakeFiles/manage.dir/all’ failed
make[1]: *** [src/CMakeFiles/manage.dir/all] Error 2
Makefile:151: recipe for target ‘all’ failed
make: *** [all] Error 2

typo error ?

A few updates on this one:

  1. I have recompiled and reinstalled everything using the latest stable version available (7.0.4)
  2. I have checked my database is ok with pragma_integrity_check
  3. I have updated & migrated my database to the latest version and with the latest nvt/cert/scap etc… fully reinstalled from scratch

Once everything up & running, I tried to run a new scan, and after a few seconds I have the same problem again:

md manage:WARNING:2019-03-19 13h08.06 utc:18091: sql_exec_internal: sqlite3_step failed: interrupted
md manage:WARNING:2019-03-19 13h08.06 utc:18091: sqlv: sql_exec_internal failed

Leading to the usual openvasmd authentication problem.

Now, with the same setup, if I restart openvasmd with option " --client-watch-interval=0", tadaaa ! Problem solved ! I can start & manage tasks as usual without any problems.

So it seems this option does the trick. But I’m not feeling comfortable with this since this option isn’t documented anywhere.

I will open a new post on that. But at least it seems we have workaround / (fix ?) for this issue.

Best

1 Like

It’s save to use that option. I am using it too for my development setup.

Some context for this parameter quoted from @tpollmeier in: Gvm-cli connections fail under openvassd load · Issue #347 · greenbone/gvmd · GitHub

It looks like openvasmd is considering the connection closed too soon.
Starting openvasmd with the option --client-watch-interval 0 may help for now.

I think we could possibly make the closed connection detection more stable by trying again
to peek at the request from the client at least once.

GREAT JOB!!!
Yes, I done a typo (wrong language lol) the right row should be:

if ( ((retries > 10) && (OPENVAS_SQLITE_SLEEP_MAX > 0)) || (ret == SQLITE_BUSY) )

But with the options is better :smiley:

Thank you all
Giovanni

Anyway I modified the /etc/init.d/openvas-manager in this way

Line34 in
[…]
[ -n “$LISTEN_ADDRESS” ] && DAEMON_ARGS=“$DAEMON_ARGS --listen=$LISTEN_ADDRESS –client-watch-interval=0
[ -n “$PORT_NUMBER” ] && DAEMON_ARGS=“$DAEMON_ARGS --port=$PORT_NUMBER –client-watch-interval=0

But it seems it doesn’t work… maybe did I wrote it in the wrong place?

You need to add the line:

DAEMON_ARGS="--client-watch-interval=0"

in your /etc/default/openvas-manager file, and then restart the process.

2 Likes

It works!
Thx

Hi Mogimbo,

I recently had the same problem again, so decided to apply your patch to sql_sqlite3.c and recompiled. After a couple of times, the same problem happens again. :frowning:

I’m considering upgrading to gvm 11 as really this bug make the product completely unreliable.

Thanks