Ubuntu Linux 19.04
Kernel version 126.96.36.199
Installation method / source:
Built from source
All works fine, no problems. The only small alteration to build defaults I did was configuring gvm with cmake “-DBACKEND=POSTGRESQL” otherwise it defaults to sqlite3 from what I could understand in the CMakeLists.txt file
Then gvmd: Syncing SCAP takes a looong time. it is this process taking its while for like 10-15 mins and then postgres itself using a whole CPU core up for a few hours.
It eventually finishes and works like a charm.
Is this expected ?
Can I tune anything in Postgres may be so it happens quicker ?
Increasing RAM and number of cores did not help. It will use either 1 core fully or 2 cores 50% each.
I read up on Postgres, which it looks like default values are more than enough for anything
Check if you XSLT Transformation takes this time. More CPU does not help be course XSLTproc can only run single CPU. A NVME and faster CPU would help.
Yeah, there is a proper server storage that helps with super fast I/O, clearly not enough.
With regards to XSLT transform taking a long time, would this happen within PostGres ? Does not sound probable to me, unless you know something I don’t. Does that work like an external function supplied by gvm ?
How could I tell if that is the bottleneck ?
This has nothing to do with the database, it´s the transformation of the XML Files.Why is this so critical ? Normally this happens within the background over night …
Sorry to hijack this thread
But I’m seeing the same on the docker images I’m building at startup.
You mentioned “Check if you XSLT Transformation takes this time”, i tried to do a strace on the process, but it’s hard to see what’s happening there
Do you have any good way to see what that process is doing?
Is it possible to debug log that process?
Kind Regards Falk
XML Parsing with xsltproc takes time and io/memory and can´t be shared across multiple CPU cores the only way to speed this up, is doing for each SCAP XML a parallel XSLT Proc and then you have to serialize the SQL Insert again.
Here are some background information:
Thanks for the good answer.