If you want to try out GVM11 src install

Hi,

As of today I saw that GVM 11 Release was released :slight_smile:

Before building docker containers for GVM11, I tried to install everything from src.
And when doing that I did a small writeup. If you want to try it out and help out with the startup scripts, here are the link https://sadsloth.net/post/install-gvm11-src_part1/ (shameless plug)

All ideas are welcome :slight_smile:

–
Regards Falk

6 Likes

Hi @falk,

You’re so fast ! I just heard about the v11 few hours ago !

I follow your tutorial instruction by instruction, I’m on Debian 9

  • libical2-dev is libical-dev on debian

  • it’s missing apt-transport-https package

  • when I dumbly copy paste your command to create the user I got this error:
    Enter the new value, or press ENTER for the default
    Full Name []: Room Number []:
    Work Phone []:
    Home Phone []:
    Other []:
    chfn: invalid room number: '_CONFIG_PATH=/opt/gvm/lib/pkgconfig:$PKG_CONFIG_PATH'
    adduser:/usr/bin/chfn gvm’ returned error code 1. Exiting.`
    It’s perfectly working, just a problem with input and output

  • There’s no git and clang-format package to install a complete version of openvas v7.0

  • Redis config is not working, but it’s maybe me, I have no real knowledge on redis… so I made my own custom configuration. I had the same problem with your previous docker of the v10, so probably linked to debian 9 ? I can elaborate if you want more information on that point.

  • gvm user has not the bin and sbin in his profile, so a simple export solve the problem to avoid using /opt/gvm/bin/greenbone-nvt-sync, /opt/gvm/sbin/gvmd etc…

  • To exit psql, it’s \q and not exit

  • On debian, you need to add nodejs:
    curl --silent --show-error https://deb.nodesource.com/gpgkey/nodesource.gpg.key | apt-key add -
    echo "deb https://deb.nodesource.com/node_8.x stretch main" | tee /etc/apt/sources.list.d/nodesource.list
    apt install nodejs -yq

I stopped at “»» HERE BE DRAGONS ««” but I’m looking forward to test the master/slave on this new version. I’ll try to finish that tonight and give a complete feedback.

As your previous version, really good job! It’s really better to create a user and store every on another lvm to separate the app from the system.

Regards,

1 Like

I cheated with the git releases of gvm11 pre releases :slight_smile:

I’ll try to make some comments with more dependencies for debian stretch.

Here I can be “more clear” where it’s possible to cut’n paste, and where every line are non “batchable” :slight_smile:

Was it possible to add those with apt before as dependencies in stretch?

Yes, that would be great.
I tried to take the config directly from the docs this time.
It adds an extra config to redis-server. And to start that service:
systemctl start redis-server@openvas.service

This should have been added with?

cp /etc/environment ~/environment.bak ;\
sudo sed -i 's|PATH="|PATH="/opt/gvm/bin:/opt/gvm/sbin:/opt/gvm/.local/bin:|g' /etc/environment ;\
sudo bash -c 'cat << EOF > /etc/ld.so.conf.d/gvm.conf
# gmv libs location
/opt/gvm/lib
EOF'

Truth, I’ll fix that (a pebkac by me) :smiley:

I’ll add those to stretch dependencies.

Those are somewhat unfinished in the guide…
They work, but lot’s of manual starting and such :slight_smile:

Thanks for the kind words!
My thoughts are that when I documenting the process for myself, I can rather give something back to the community. In the way I can :slight_smile:

–
Regards Falk

2 Likes

Here I can be “more clear” where it’s possible to cut’n paste, and where every line are non “batchable” :slightly_smiling_face:

Yes, I played the dumb, as I like to ask some friend to do that when I create a documentation, to be sure that anyone can use it without having any issue.

I’ll try to make some comments with more dependencies for debian stretch.

Was it possible to add those with apt before as dependencies in stretch?

Yes, a simple apt install git clang-format apt-transport-https libical-dev solve the problem.

Yes, that would be great.
I tried to take the config directly from the docs this time.
It adds an extra config to redis-server. And to start that service: systemctl start redis-server@openvas.service

I’m aware of the @ symbol that makes you start a specific instance of a service, but there it was not working for me by simply printing: Failed to start redis-server@openvas.service: Unit redis-server@openvas.service not found.

As I told you I have no real knowledge on redis… So to test a bit the configuration file, I simply move redis-openvas.conf as redis.conf, and there I had the same that I got with your tutorial on GVM10.

When I start the service i got an error, which come out this way with systemctl status redis-server:
systemd[1]: redis-server.service: Unit entered failed state.
systemd[1]: redis-server.service: Failed with result 'resources'.
systemd[1]: redis-server.service: Service hold-off time over, scheduling restart.
systemd[1]: Stopped Advanced key-value store.
systemd[1]: redis-server.service: Start request repeated too quickly.
systemd[1]: Failed to start Advanced key-value store.
systemd[1]: redis-server.service: Unit entered failed state.
systemd[1]: redis-server.service: Failed with result 'resources'.

So I checked on the interne and found this on stackoverflow: >For me the solution was to give access to the config file (read) and the log file (read-write) to user redis.

chown -R redis:redis /var/log/redis
chmod -R u+rwX,g+rwX,u+rx /var/log/redis
chmod +r /etc/redis/redis.conf

Seems like this guy is right, because after that it’s working well. Still, I cannot run a specific instance, I have to look it up more about redis and how to run a specific instance.

This should have been added with?

On a fresh debian 9, you don’t have any /etc/environment file, so the sed don’t do anything. Seems like /etc/environment is still working tho, but we usually use /etc/profile for the path.

Those are somewhat unfinished in the guide…
They work, but lots of manual starting and such :slightly_smiling_face:

You probably should come up with some services, but I don’t know how you can do that with ospd.

By the way, if you try service, on my previous version with psql, I had to specify in the service that gvmd should start after psql. (It was working without this on sqlite)

All ideas are welcome :slightly_smiling_face:

I have a concern about docker version of a master/slave setup. if you deploy them at the same time or in a reduced time interval, one of them will be temporary ban by greenbone and therefore will not be able to update its NVTs.

So I thought about a rsync between the master and all the slave, I’m currently testing the solution on my setup, I’ll give you a feedback when it’s done.

Another area of improvement that I’ll work when I have more time is the NVT, as you may know there is two different feeds from Greenbone, GCF and GSF.

Unfortunately, we cannot purchase the GSF without having an appliance of greenbone with their OS. So to fix this, I found out this and I’d like to test it, we’ll see if it’s still working.

My thoughts are that when I documenting the process for myself, I can rather give something back to the community. In the way I can :slightly_smiling_face:

That’s the spirit, I hope I can give you more feedback about the improvement I planned to have soon ! Keep it up !

Regards,

1 Like

Just a hint: ospd-openvas/config at ospd-openvas-1.0 · greenbone/ospd-openvas · GitHub

1 Like

I highly advise not to use an external feed for production. There is no guarantee this feed works as expected with different GVM versions. Please only use the GCF or the GSF.

Just a hint: ospd-openvas/config at ospd-openvas-1.0 · greenbone/ospd-openvas · GitHub

Thanks for the link, I will look when I tackle gvm11 in more depth, I didn’t need to study how it works with previous versions.

I highly advise not to use an external feed for production. There is no guarantee this feed works as expected with different GVM versions. Please only use the GCF or the GSF.

It’s more curiosity about what the possibilities are, to see how to make my own nasl later on for very specific systems.

Please only use the GCF or the GSF.

Are you now selling GSF separately ?

The GSF and GCF feeds are the only feeds supported. Currently the GSF is only available in conjunction with a Greenbone product.

1 Like

Now, finally, even I have gotten it to work :slight_smile:
Page is updated with the “what I think” is a working install…

All comments are welcome

–
Regards Falk

1 Like

Hello @falk, and thank your for your guide.

I have a working install mostly based on your guide, but i notice the “Syncing SCAP” phase is taking 18+ hours to complete.

There are no errors and i am able to run scans now, but this process I have read can “take a long time” but no where do i see any mention of overnight, or 18+ hours.

My postgresql and gvmd processes bounce around near 100% CPU usage, and the XML files are being built. Just very slowly.

Has anyone else run into this long running SCAP sync/ build of the scap xml files? Is this 18+ hour process normal for a source built GVM install?

Notice the many hours between each file being finished:

$ sudo -H -u greenbone tail -f /opt/install/gvm11/var/log/gvm/gvmd.log
md manage: INFO:2019-11-06 21h38.40 utc:1766: Updating /opt/install/gvm11/var/lib/gvm/scap-data/nvdcve-2.0-2006.xml
md manage: INFO:2019-11-06 21h40.20 utc:1766: Updating /opt/install/gvm11/var/lib/gvm/scap-data/nvdcve-2.0-2014.xml
md manage: INFO:2019-11-06 22h27.53 utc:1766: Updating /opt/install/gvm11/var/lib/gvm/scap-data/nvdcve-2.0-2012.xml
md manage: INFO:2019-11-06 23h05.05 utc:1766: Updating /opt/install/gvm11/var/lib/gvm/scap-data/nvdcve-2.0-2016.xml
md manage: INFO:2019-11-07 08h17.45 utc:1766: Updating /opt/install/gvm11/var/lib/gvm/scap-data/nvdcve-2.0-2019.xml
md manage: INFO:2019-11-07 09h06.34 utc:1766: Updating /opt/install/gvm11/var/lib/gvm/scap-data/nvdcve-2.0-2013.xml
md manage: INFO:2019-11-07 10h11.00 utc:1766: Updating /opt/install/gvm11/var/lib/gvm/scap-data/nvdcve-2.0-2003.xml
md manage: INFO:2019-11-07 10h12.35 utc:1766: Updating /opt/install/gvm11/var/lib/gvm/scap-data/nvdcve-2.0-2009.xml
md manage: INFO:2019-11-07 10h23.44 utc:1766: Updating /opt/install/gvm11/var/lib/gvm/scap-data/nvdcve-2.0-2011.xml
md manage: INFO:2019-11-07 15h16.35 utc:1766: Updating /opt/install/gvm11/var/lib/gvm/scap-data/nvdcve-2.0-2015.xml
md manage: INFO:2019-11-07 16h58.26 utc:1766: Updating /opt/install/gvm11/var/lib/gvm/scap-data/nvdcve-2.0-2010.xml
md manage: INFO:2019-11-07 17h34.45 utc:1766: Updating /opt/install/gvm11/var/lib/gvm/scap-data/nvdcve-2.0-2004.xml
md manage: INFO:2019-11-07 17h36.00 utc:1766: Updating /opt/install/gvm11/var/lib/gvm/scap-data/nvdcve-2.0-2005.xml
md manage: INFO:2019-11-07 17h38.10 utc:1766: Updating /opt/install/gvm11/var/lib/gvm/scap-data/nvdcve-2.0-2007.xml
md manage: INFO:2019-11-07 17h44.42 utc:1766: Updating /opt/install/gvm11/var/lib/gvm/scap-data/nvdcve-2.0-2018.xml

1 Like

I just found a related bug to my issue has already been recorded over in the github repo.
Looks like a fix is already in the works.

1 Like

I just installed a GCE and perfomed a initital feed update with todays feed version (SCAP: 20191107T1401). It finished within 1h. Could you please try again and report if this issue still exists?

2 Likes

The bug report listed above was updated today to note that the upstream feed had some changes that caused this problem, and that the feed data itself should now be fixed to eliminate this 24+ hour sync issue without needing to upgrade or change GVM itself.

Important to note that even with the bug fixed, the initial SCAP sync can still take 30+ mins depending on hardware and bandwidth.

I’m re-syncing the SCAP data now, will report back on how long it takes on my 4th gen i5 system so people can compare what is “normal” for this process down the road.

1 Like

I just re-ran the scap update, took about 90 minutes on the same system that took 25 hours to do the update a few days ago with the “bad feed”.

Thanks to the greenbone dev’s for getting this fixed up so quickly!

2 Likes

Where can I find a gvm11 (working) docker container?

Thanks

Hi,

I don’t know if anyone else has one up and running yet.
But I’m haven’t built “mine” yet, it’s not going so great :slight_smile:

–
Regards Falk

Hi,

during the last week I was playing around with setting up a gvm11 installation inside a docker container. My “hacky” version can be found here: https://github.com/carlstegmann/snippets/tree/master/gvm11_inside_docker do not use this in production!
Also it is sometimes a bit buggy, I guess this is simply because of my way not running ospd as regular user and so it seems the /tmp/ospd.sock will loose the configured permissions, but it works for me :wink::sunglasses:

Cheers,
Carl