Issues with CPU usage and updates

I’ve built GVM-11 (release gvmd-9.0) and also GVM-11 (Master gvmd-9.0.1/2) from sources on both ubuntu 18.04 and 20.04 and also centos 8. IN all instances it completly builds but when I start it up I have two issues with it. Thats why mI rebuilt it using different dists and versions.

  1. I never get any Oval definitions in gui. Everything else is there, nvt, cve, cert, cpe, dfn…but oval is always zero.
  2. I can see in the logs that there seems to be an issues with gettings ovals from the feed, which makes it try to import it once every 5 minutes hence the server is always on 100% cpu on one core. The following is the logs from one of these builds…I thought this would finish but after 48 hours on a Xeon gold 6250 x 2, I gave up and had a look in logs…this is it:

This is GVMD.log

md manage: INFO:2020-06-02 19h05.45 utc:115128: Updating /opt/gvm/var/lib/gvm/scap-data/nvdcve-2.0-2011.xml
md manage: INFO:2020-06-02 19h05.46 utc:115128: Updating /opt/gvm/var/lib/gvm/scap-data/nvdcve-2.0-2019.xml
md manage: INFO:2020-06-02 19h05.46 utc:115128: Updating /opt/gvm/var/lib/gvm/scap-data/nvdcve-2.0-2017.xml
md manage: INFO:2020-06-02 19h05.47 utc:115128: Updating /opt/gvm/var/lib/gvm/scap-data/nvdcve-2.0-2006.xml
md manage: INFO:2020-06-02 19h05.47 utc:115128: Updating /opt/gvm/var/lib/gvm/scap-data/nvdcve-2.0-2014.xml
md manage: INFO:2020-06-02 19h05.47 utc:115128: Updating /opt/gvm/var/lib/gvm/scap-data/nvdcve-2.0-2004.xml
md manage: INFO:2020-06-02 19h05.48 utc:115128: Updating /opt/gvm/var/lib/gvm/scap-data/nvdcve-2.0-2010.xml
md manage: INFO:2020-06-02 19h05.48 utc:115128: Updating /opt/gvm/var/lib/gvm/scap-data/nvdcve-2.0-2015.xml
md manage: INFO:2020-06-02 19h05.48 utc:115128: Updating /opt/gvm/var/lib/gvm/scap-data/nvdcve-2.0-2008.xml
md manage: INFO:2020-06-02 19h05.48 utc:115128: Updating /opt/gvm/var/lib/gvm/scap-data/nvdcve-2.0-2003.xml
md manage: INFO:2020-06-02 19h05.48 utc:115128: Updating /opt/gvm/var/lib/gvm/scap-data/nvdcve-2.0-2016.xml
md manage: INFO:2020-06-02 19h05.49 utc:115128: Updating /opt/gvm/var/lib/gvm/scap-data/nvdcve-2.0-2020.xml
md manage: INFO:2020-06-02 19h05.49 utc:115128: Updating /opt/gvm/var/lib/gvm/scap-data/nvdcve-2.0-2013.xml
md manage: INFO:2020-06-02 19h05.49 utc:115128: Updating /opt/gvm/var/lib/gvm/scap-data/nvdcve-2.0-2018.xml
md manage:WARNING:2020-06-02 19h05.49 utc:115168: manage_update_nvt_cache_osp: a feed sync is already running
md manage: INFO:2020-06-02 19h05.50 utc:115128: Updating /opt/gvm/var/lib/gvm/scap-data/nvdcve-2.0-2012.xml
md manage: INFO:2020-06-02 19h05.50 utc:115128: Updating /opt/gvm/var/lib/gvm/scap-data/nvdcve-2.0-2007.xml
md manage: INFO:2020-06-02 19h05.50 utc:115128: Updating /opt/gvm/var/lib/gvm/scap-data/nvdcve-2.0-2002.xml
md manage: INFO:2020-06-02 19h05.51 utc:115128: Updating /opt/gvm/var/lib/gvm/scap-data/nvdcve-2.0-2005.xml
md manage:WARNING:2020-06-02 19h06.00 utc:115174: manage_update_nvt_cache_osp: a feed sync is already running
md manage:WARNING:2020-06-02 19h06.10 utc:115180: manage_update_nvt_cache_osp: a feed sync is already running
md manage:WARNING:2020-06-02 19h06.20 utc:115185: manage_update_nvt_cache_osp: a feed sync is already running
md manage:WARNING:2020-06-02 19h06.30 utc:115194: manage_update_nvt_cache_osp: a feed sync is already running
md manage:WARNING:2020-06-02 19h06.40 utc:115200: manage_update_nvt_cache_osp: a feed sync is already running
md manage:WARNING:2020-06-02 19h06.50 utc:115207: manage_update_nvt_cache_osp: a feed sync is already running
md manage:WARNING:2020-06-02 19h07.00 utc:115213: manage_update_nvt_cache_osp: a feed sync is already running
md manage:WARNING:2020-06-02 19h07.10 utc:115220: manage_update_nvt_cache_osp: a feed sync is already running
md manage:WARNING:2020-06-02 19h07.20 utc:115226: manage_update_nvt_cache_osp: a feed sync is already running
md manage:WARNING:2020-06-02 19h07.30 utc:115234: manage_update_nvt_cache_osp: a feed sync is already running
md manage:WARNING:2020-06-02 19h07.40 utc:115240: manage_update_nvt_cache_osp: a feed sync is already running
md manage:WARNING:2020-06-02 19h07.50 utc:115247: manage_update_nvt_cache_osp: a feed sync is already running
md manage:WARNING:2020-06-02 19h08.00 utc:115253: manage_update_nvt_cache_osp: a feed sync is already running
md manage:WARNING:2020-06-02 19h08.10 utc:115260: manage_update_nvt_cache_osp: a feed sync is already running
md manage:WARNING:2020-06-02 19h08.20 utc:115266: manage_update_nvt_cache_osp: a feed sync is already running
md manage:WARNING:2020-06-02 19h08.30 utc:115273: manage_update_nvt_cache_osp: a feed sync is already running
md manage:WARNING:2020-06-02 19h08.49 utc:115282: manage_update_nvt_cache_osp: a feed sync is already running
md manage: INFO:2020-06-02 19h08.49 utc:115281: update_scap: Updating data from feed
md manage: INFO:2020-06-02 19h08.49 utc:115281: Updating CPEs
md manage:WARNING:2020-06-02 19h08.59 utc:115291: manage_update_nvt_cache_osp: a feed sync is already running
md manage:WARNING:2020-06-02 19h09.09 utc:115298: manage_update_nvt_cache_osp: a feed sync is already running
md manage: INFO:2020-06-02 19h09.15 utc:115281: Updating /opt/gvm/var/lib/gvm/scap-data/nvdcve-2.0-2009.xml
md manage: INFO:2020-06-02 19h09.15 utc:115281: Updating /opt/gvm/var/lib/gvm/scap-data/nvdcve-2.0-2011.xml
md manage: INFO:2020-06-02 19h09.15 utc:115281: Updating /opt/gvm/var/lib/gvm/scap-data/nvdcve-2.0-2019.xml
md manage: INFO:2020-06-02 19h09.16 utc:115281: Updating /opt/gvm/var/lib/gvm/scap-data/nvdcve-2.0-2017.xml
md manage: INFO:2020-06-02 19h09.16 utc:115281: Updating /opt/gvm/var/lib/gvm/scap-data/nvdcve-2.0-2006.xml
md manage: INFO:2020-06-02 19h09.17 utc:115281: Updating /opt/gvm/var/lib/gvm/scap-data/nvdcve-2.0-2014.xml
md manage: INFO:2020-06-02 19h09.17 utc:115281: Updating /opt/gvm/var/lib/gvm/scap-data/nvdcve-2.0-2004.xml
md manage: INFO:2020-06-02 19h09.17 utc:115281: Updating /opt/gvm/var/lib/gvm/scap-data/nvdcve-2.0-2010.xml
md manage: INFO:2020-06-02 19h09.17 utc:115281: Updating /opt/gvm/var/lib/gvm/scap-data/nvdcve-2.0-2015.xml
md manage: INFO:2020-06-02 19h09.18 utc:115281: Updating /opt/gvm/var/lib/gvm/scap-data/nvdcve-2.0-2008.xml
md manage: INFO:2020-06-02 19h09.18 utc:115281: Updating /opt/gvm/var/lib/gvm/scap-data/nvdcve-2.0-2003.xml
md manage: INFO:2020-06-02 19h09.18 utc:115281: Updating /opt/gvm/var/lib/gvm/scap-data/nvdcve-2.0-2016.xml
md manage: INFO:2020-06-02 19h09.18 utc:115281: Updating /opt/gvm/var/lib/gvm/scap-data/nvdcve-2.0-2020.xml
md manage: INFO:2020-06-02 19h09.19 utc:115281: Updating /opt/gvm/var/lib/gvm/scap-data/nvdcve-2.0-2013.xml
md manage: INFO:2020-06-02 19h09.19 utc:115281: Updating /opt/gvm/var/lib/gvm/scap-data/nvdcve-2.0-2018.xml
md manage:WARNING:2020-06-02 19h09.19 utc:115319: manage_update_nvt_cache_osp: a feed sync is already running
md manage: INFO:2020-06-02 19h09.19 utc:115281: Updating /opt/gvm/var/lib/gvm/scap-data/nvdcve-2.0-2012.xml
md manage: INFO:2020-06-02 19h09.20 utc:115281: Updating /opt/gvm/var/lib/gvm/scap-data/nvdcve-2.0-2007.xml
md manage: INFO:2020-06-02 19h09.20 utc:115281: Updating /opt/gvm/var/lib/gvm/scap-data/nvdcve-2.0-2002.xml
md manage: INFO:2020-06-02 19h09.20 utc:115281: Updating /opt/gvm/var/lib/gvm/scap-data/nvdcve-2.0-2005.xml
md manage:WARNING:2020-06-02 19h09.29 utc:115326: manage_update_nvt_cache_osp: a feed sync is already running
md manage:WARNING:2020-06-02 19h09.39 utc:115331: manage_update_nvt_cache_osp: a feed sync is already running
md manage:WARNING:2020-06-02 19h09.49 utc:115339: manage_update_nvt_cache_osp: a feed sync is already running
md manage:WARNING:2020-06-02 19h09.59 utc:115346: manage_update_nvt_cache_osp: a feed sync is already running
md manage:WARNING:2020-06-02 19h10.09 utc:115353: manage_update_nvt_cache_osp: a feed sync is already running
md manage:WARNING:2020-06-02 19h10.19 utc:115359: manage_update_nvt_cache_osp: a feed sync is already running
md manage:WARNING:2020-06-02 19h10.29 utc:115366: manage_update_nvt_cache_osp: a feed sync is already running
md manage:WARNING:2020-06-02 19h10.39 utc:115372: manage_update_nvt_cache_osp: a feed sync is already running
md manage:WARNING:2020-06-02 19h10.49 utc:115379: manage_update_nvt_cache_osp: a feed sync is already running
md manage:WARNING:2020-06-02 19h10.59 utc:115385: manage_update_nvt_cache_osp: a feed sync is already running
md manage:WARNING:2020-06-02 19h11.09 utc:115391: manage_update_nvt_cache_osp: a feed sync is already running
md manage:WARNING:2020-06-02 19h11.19 utc:115399: manage_update_nvt_cache_osp: a feed sync is already running
md manage:WARNING:2020-06-02 19h11.29 utc:115406: manage_update_nvt_cache_osp: a feed sync is already running
md manage:WARNING:2020-06-02 19h11.39 utc:115414: manage_update_nvt_cache_osp: a feed sync is already running
md manage:WARNING:2020-06-02 19h11.49 utc:115536: manage_update_nvt_cache_osp: a feed sync is already running
md manage:WARNING:2020-06-02 19h11.59 utc:115542: manage_update_nvt_cache_osp: a feed sync is already running
md manage:WARNING:2020-06-02 19h12.18 utc:115550: manage_update_nvt_cache_osp: a feed sync is already running
md manage: INFO:2020-06-02 19h12.18 utc:115549: update_scap: Updating data from feed
md manage: INFO:2020-06-02 19h12.18 utc:115549: Updating CPEs
md manage:WARNING:2020-06-02 19h12.28 utc:115560: manage_update_nvt_cache_osp: a feed sync is already running
md manage:WARNING:2020-06-02 19h12.38 utc:115567: manage_update_nvt_cache_osp: a feed sync is already running
md manage: INFO:2020-06-02 19h12.44 utc:115549: Updating /opt/gvm/var/lib/gvm/scap-data/nvdcve-2.0-2009.xml
md manage: INFO:2020-06-02 19h12.44 utc:115549: Updating /opt/gvm/var/lib/gvm/scap-data/nvdcve-2.0-2011.xml
md manage: INFO:2020-06-02 19h12.45 utc:115549: Updating /opt/gvm/var/lib/gvm/scap-data/nvdcve-2.0-2019.xml
md manage: INFO:2020-06-02 19h12.45 utc:115549: Updating /opt/gvm/var/lib/gvm/scap-data/nvdcve-2.0-2017.xml
md manage: INFO:2020-06-02 19h12.46 utc:115549: Updating /opt/gvm/var/lib/gvm/scap-data/nvdcve-2.0-2006.xml
md manage: INFO:2020-06-02 19h12.46 utc:115549: Updating /opt/gvm/var/lib/gvm/scap-data/nvdcve-2.0-2014.xml
md manage: INFO:2020-06-02 19h12.46 utc:115549: Updating /opt/gvm/var/lib/gvm/scap-data/nvdcve-2.0-2004.xml
md manage: INFO:2020-06-02 19h12.47 utc:115549: Updating /opt/gvm/var/lib/gvm/scap-data/nvdcve-2.0-2010.xml
md manage: INFO:2020-06-02 19h12.47 utc:115549: Updating /opt/gvm/var/lib/gvm/scap-data/nvdcve-2.0-2015.xml
md manage: INFO:2020-06-02 19h12.47 utc:115549: Updating /opt/gvm/var/lib/gvm/scap-data/nvdcve-2.0-2008.xml
md manage: INFO:2020-06-02 19h12.47 utc:115549: Updating /opt/gvm/var/lib/gvm/scap-data/nvdcve-2.0-2003.xml
md manage: INFO:2020-06-02 19h12.47 utc:115549: Updating /opt/gvm/var/lib/gvm/scap-data/nvdcve-2.0-2016.xml
md manage: INFO:2020-06-02 19h12.48 utc:115549: Updating /opt/gvm/var/lib/gvm/scap-data/nvdcve-2.0-2020.xml
md manage: INFO:2020-06-02 19h12.48 utc:115549: Updating /opt/gvm/var/lib/gvm/scap-data/nvdcve-2.0-2013.xml
md manage: INFO:2020-06-02 19h12.48 utc:115549: Updating /opt/gvm/var/lib/gvm/scap-data/nvdcve-2.0-2018.xml
md manage:WARNING:2020-06-02 19h12.48 utc:115589: manage_update_nvt_cache_osp: a feed sync is already running
md manage: INFO:2020-06-02 19h12.49 utc:115549: Updating /opt/gvm/var/lib/gvm/scap-data/nvdcve-2.0-2012.xml
md manage: INFO:2020-06-02 19h12.49 utc:115549: Updating /opt/gvm/var/lib/gvm/scap-data/nvdcve-2.0-2007.xml
md manage: INFO:2020-06-02 19h12.49 utc:115549: Updating /opt/gvm/var/lib/gvm/scap-data/nvdcve-2.0-2002.xml
md manage: INFO:2020-06-02 19h12.50 utc:115549: Updating /opt/gvm/var/lib/gvm/scap-data/nvdcve-2.0-2005.xml
md manage:WARNING:2020-06-02 19h12.58 utc:115595: manage_update_nvt_cache_osp: a feed sync is already running
md manage:WARNING:2020-06-02 19h13.08 utc:115603: manage_update_nvt_cache_osp: a feed sync is already running
md manage:WARNING:2020-06-02 19h13.18 utc:115607: manage_update_nvt_cache_osp: a feed sync is already running
md manage:WARNING:2020-06-02 19h13.28 utc:115615: manage_update_nvt_cache_osp: a feed sync is already running
md manage:WARNING:2020-06-02 19h13.38 utc:115621: manage_update_nvt_cache_osp: a feed sync is already running
md manage:WARNING:2020-06-02 19h13.48 utc:115628: manage_update_nvt_cache_osp: a feed sync is already running
md manage:WARNING:2020-06-02 19h13.58 utc:115634: manage_update_nvt_cache_osp: a feed sync is already running
md manage:WARNING:2020-06-02 19h14.08 utc:115641: manage_update_nvt_cache_osp: a feed sync is already running
md manage:WARNING:2020-06-02 19h14.19 utc:115647: manage_update_nvt_cache_osp: a feed sync is already running
md manage:WARNING:2020-06-02 19h14.29 utc:115654: manage_update_nvt_cache_osp: a feed sync is already running
md manage:WARNING:2020-06-02 19h14.39 utc:115660: manage_update_nvt_cache_osp: a feed sync is already running
md manage:WARNING:2020-06-02 19h14.49 utc:115667: manage_update_nvt_cache_osp: a feed sync is already running
md manage:WARNING:2020-06-02 19h14.59 utc:115676: manage_update_nvt_cache_osp: a feed sync is already running
md manage:WARNING:2020-06-02 19h15.09 utc:115686: manage_update_nvt_cache_osp: a feed sync is already running
md manage:WARNING:2020-06-02 19h15.19 utc:115695: manage_update_nvt_cache_osp: a feed sync is already running
md manage:WARNING:2020-06-02 19h15.29 utc:115701: manage_update_nvt_cache_osp: a feed sync is already running
md manage:WARNING:2020-06-02 19h15.48 utc:115719: manage_update_nvt_cache_osp: a feed sync is already running
md manage: INFO:2020-06-02 19h15.48 utc:115718: update_scap: Updating data from feed
md manage: INFO:2020-06-02 19h15.48 utc:115718: Updating CPEs
md manage:WARNING:2020-06-02 19h15.58 utc:115736: manage_update_nvt_cache_osp: a feed sync is already running
md manage:WARNING:2020-06-02 19h16.08 utc:115747: manage_update_nvt_cache_osp: a feed sync is already running

You need at least two cores, and i suggest to use very fast storage like NVME or ram disk for the IO intensive tasks.

it has got 16 cores @ 4.5 Ghz (VM) and 32 Gb ram (VM can add more). It uses 6 x 3.84 TB nvme. It’s a supermicro server at work.

Then is should take minutes and not hours, either your setup or your environment must be erroneous.

The xml_split recommendation in newer GVM versions might help to speed up the sync process:

It will just keep trying if there’s a problem. The log doesn’t show what the problem is, so it’s hard to diagnose. Perhaps just remove the oval dir from your feed?

I think those “sync is already running” warnings are because we switched to a single lockfile recently. Should not be a problem.

I have the XML-twig-tools installed which includes xml_split. I will try to delete synced oval defs and see what happens.

This is odd, I did this before (delete all feed / scap-data and resync). I did the same today and it works on all of the VMs, Ubuntu 1804, 2004, centos 8. All works. I remember seeing some XML-error in a log a few days back. Can’t remember which one tho, but it pointed to malformed feed xml.