Please be aware that only one session at the time is possible, if you are not direct connected to the internet it is mostly a other GVM system, or a network device that might block you from syncing. NAT-Gates and firewalls trend to keep sessions alive, even if your side is down
After hours and hours of going through loads and loads of this I have found that all services can be green yet the web interface not be ready, scanner failing verification etc. Here is my write-up regarding my experience with this application in regard to feed updates. Please feel free to expand on this further. This software is really good and I have contributed to many open source projects so a ton of time goes in to building and maintaining it so thank you for providing it!!
Update Process (Paths based on Debian build from source):
Update in this order:
greenbone-nvt-sync
Need to wait for this to complete. If it dies in the middle of the sync a .lock file will be left in the /opt/gvm/var/run/*.lock this lock file will need to be removed and then run the greenbone-nvt-sync command again. Failure to do this and moving on to the next sync will certainly break things!
greenbone-certdata-sync
Follows nvt sync and usually completes without error. Again check for the presence of a .lock file and run the command again until it completes
greenbone-scapdata-sync
Follow the same guidelines as above
When these commands are complete check the gvmd service status
systemctl status gvmd
Output similar to the following will be shown:
â gvmd.service - Job that runs the gvm daemon
Loaded: loaded (/etc/systemd/system/gvmd.service; enabled; vendor preset: ena
Active: active (running) since Mon 2020-06-08 20:15:50 CDT; 47min ago
Docs: man:gvm
Process: 661 ExecStart=/opt/gvm/sbin/gvmd --osp-vt-update=/opt/gvm/var/run/osp
Main PID: 667 (gvmd)
Tasks: 3 (limit: 4689)
Memory: 226.9M
CGroup: /system.slice/gvmd.service
ââ 667 gvmd: Waiting for incoming connections
ââ6060 gvmd: Reloading NVTs
ââ6062 gvmd: OSP: Updating NVT cache
Until the service is done reloading operations it’s best not to try to run scans or disrupt it in any way. The updating takes a bit so when you do this it’s probably best to let it run through the evening then use it in the AM to scan.
A finished output looks similar to the following:
â gvmd.service - Job that runs the gvm daemon
Loaded: loaded (/etc/systemd/system/gvmd.service; enabled; vendor preset: ena
Active: active (running) since Mon 2020-06-08 20:15:50 CDT; 1h 0min ago
Docs: man:gvm
Process: 661 ExecStart=/opt/gvm/sbin/gvmd --osp-vt-update=/opt/gvm/var/run/osp
Main PID: 667 (gvmd)
Tasks: 1 (limit: 4689)
Memory: 224.5M
CGroup: /system.slice/gvmd.service
ââ667 gvmd: Waiting for incoming connections
The following services should also be checked as well for a running state:
systemctl status gsad
systemctl status ospd-openvas
If all three services are running, the system should open and function correctly.
-If the feeds get hosed up the software will not run the web browser and you are rolling back (VM) or restoring from a backup. It’s one of the things you have to deal with. The software is free after all.
-D-
If someone is having good luck with running updates thorough cron I would be interested on how you are handling the update failures when they occur.
@Lukas - thanks for that info. I’m reasonably certain there weren’t other sync processes running from other systems. It’s possible that a NAT gateway was holding open a stale session, but I can test that by simply restarting my gateway and the Greenbone virtual machine.
-
Update: restarted gateway and VM. No joy. However, I think I’m narrowing in on my problem, and perhaps this can help someone else. When I manually run rsync from the command line on my physical machine, I am able to connect and retrieve files with no issue. However, on the Greenbone VM, which is in VirtualBox, I’m having issues. It could be the bridged connection, or something like that causing issues. I’ll test some more and report back.
-
Update 2: the failures seemed to be random in nature, so I was suspecting network-ey things as @Lukas suggested. I added some arbitrary sleep statements into the greenbone-nvt-sync script to give it time in between rsync commands to the feed update server. This seems to work all the time. So, perhaps it is some stale session in a NAT gateway or firewall, but I can’t prove it.
-
Update 3: Arbitrary sleep statements seem to make it work every time. Taking them out immediately caused a failure. Putting them back in, and it works again. So my suspicion is that something is causing it to look like it’s two sessions, when it’s only one script running but with rsync commands really close together in timing. When I spread out the timing of the multiple rsync commands, it works. I hate arbitrary sleep statements, but it works.
How did I get here? I noticed that the first “receiving incremental file list” message succeeded, and subsequent ones in the script failed. Why would one work and another fail in the same script? Probably because there’s a stale session caused by NAT, firewall, or VirtualBox.
Not sure what the issue is yet. From the VM, it’s working right now. But a few minutes ago it wasn’t. No substantial changes.
- Final update: with arbitrary sleep statements in greenbone-scapdata-sync, greenbone-certdata-sync, and greenbone-nvt-sync after each and every rsync command, it works every time.
It’s a very kludgy workaround and I’m not proud of it. But it gets around the NAT/firewall/VM issues or whatever is causing stale session info on the server side.
Care to share the solution?
Something like this. Arbitrary sleep statements after each call to rsync. It’s definitely not pretty. I did this in all three sync scripts (greenbone-nvt-sync, greenbone-certdata-sync, greenbone-scapdata-sync). Let me know if it works for you as well.
#!/bin/sh
# Copyright (C) 2009-2020 Greenbone Networks GmbH
#
# SPDX-License-Identifier: GPL-2.0-or-later
#
# This program is free software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 2
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA.
# This script updates the local Network Vulnerability Tests (NVTs) from the
# Greenbone Security Feed (GSF) or the Greenbone Community Feed (GCF).
VERSION=@OPENVAS_VERSION@
# SETTINGS
# ========
# PRIVATE_SUBDIR defines a subdirectory of the NVT directory that is excluded
# from the feed sync. This is where to place your own NVTs.
if [ -z "$PRIVATE_SUBDIR" ]
then
PRIVATE_SUBDIR="private"
fi
# RSYNC_DELETE controls whether files which are not part of the repository will
# be removed from the local directory after synchronization. The default value
# for this setting is
# "--delete --exclude \"$PRIVATE_SUBDIR/\"",
# which means that files which are not part of the feed or private directory
# will be deleted.
RSYNC_DELETE="--delete --exclude $PRIVATE_SUBDIR/"
# RSYNC_SSH_OPTS contains options which should be passed to ssh for the rsync
# connection to the repository.
RSYNC_SSH_OPTS="-o \"UserKnownHostsFile=/dev/null\" -o \"StrictHostKeyChecking=no\""
# RSYNC_COMPRESS specifies the compression level to use for the rsync connection.
RSYNC_COMPRESS="--compress-level=9"
# RSYNC_CHMOD specifies the permissions to chmod the files to.
RSYNC_CHMOD="--perms --chmod=Fugo+r,Fug+w,Dugo-s,Dugo+rx,Dug+w"
# Verbosity flag for rsync. "-q" means a quiet rsync, "-v" a verbose rsync.
RSYNC_VERBOSE="-q"
# RSYNC_OPTIONS controls the general parameters for the rsync connection.
RSYNC_OPTIONS="--links --times --omit-dir-times $RSYNC_VERBOSE --recursive --partial --progress"
# Script and feed information which will be made available to user through
# command line options and automated tools.
# Script name which will be used for logging
SCRIPT_NAME="greenbone-nvt-sync"
# Result of selftest () is stored here. If it is not 0, the selftest has failed
# and the sync script is unlikely to work.
SELFTEST_FAIL=0
# Port to use for synchronization. Default value is 24.
PORT=24
# Directory where the OpenVAS configuration is located
OPENVAS_SYSCONF_DIR="@OPENVAS_SYSCONF_DIR@"
# Directory where the feed update lock file will be placed.
OPENVAS_RUN_DIR="@OPENVAS_RUN_DIR@"
# Location of the GSF Access Key
ACCESS_KEY="@GVM_ACCESS_KEY_DIR@/gsf-access-key"
# If ENABLED is set to 0, the sync script will not perform a synchronization.
ENABLED=1
# LOG_CMD defines the command to use for logging. To have logger log to stderr
# as well as syslog, add "-s" here. The logging facility is checked. In case of error
# all will be logged in the standard error and the socket error check will be
# disabled.
LOG_CMD="logger -t $SCRIPT_NAME"
check_logger () {
logger --socket-error=on -p daemon.info -t $SCRIPT_NAME "Checking logger" --no-act 1>/dev/null 2>&1
if [ $? -gt 0 ]
then
LOG_CMD="logger --socket-error=off -s -t $SCRIPT_NAME"
$LOG_CMD -p daemon.warning "The log facility is not working as expected. All messages will be written to the standard error stream."
fi
}
check_logger
# Source configuration file if it is readable
[ -r $OPENVAS_SYSCONF_DIR/greenbone-nvt-sync.conf ] && . $OPENVAS_SYSCONF_DIR/greenbone-nvt-sync.conf
# NVT_DIR is the place where the NVTs are located.
if [ -z "$NVT_DIR" ]
then
NVT_DIR="@OPENVAS_NVT_DIR@"
fi
log_write () {
$LOG_CMD -p daemon.notice $1
}
log_debug () {
$LOG_CMD -p daemon.debug "$1"
}
log_info () {
$LOG_CMD -p daemon.info "$1"
}
log_notice () {
$LOG_CMD -p daemon.notice "$1"
}
log_warning () {
$LOG_CMD -p daemon.warning "$1"
}
log_err () {
$LOG_CMD -p daemon.err "$1"
}
stderr_write ()
{
echo "$1" > /dev/stderr
}
# Read the general information about the feed origin from
# the file "plugin_feed_info.inc" inside the feed directory.
get_feed_info ()
{
INFOFILE="$NVT_DIR/plugin_feed_info.inc"
if [ -r $INFOFILE ] ; then
FEED_VERSION=`grep PLUGIN_SET $INFOFILE | sed -e 's/[^0-9]//g'`
FEED_NAME=`awk -F\" '/PLUGIN_FEED/ { print $2 }' $INFOFILE`
FEED_VENDOR=`awk -F\" '/FEED_VENDOR/ { print $2 }' $INFOFILE`
FEED_HOME=`awk -F\" '/FEED_HOME/ { print $2 }' $INFOFILE`
FEED_PRESENT=1
else
FEED_PRESENT=0
fi
if [ -z "$FEED_NAME" ] ; then
FEED_NAME="Unidentified Feed"
fi
if [ -z "$FEED_VENDOR" ] ; then
FEED_VENDOR="Unidentified Vendor"
fi
if [ -z "$FEED_HOME" ] ; then
FEED_HOME="Unidentified Feed Homepage"
fi
}
# Prevent that root executes this script
if [ "`id -u`" -eq "0" ]
then
stderr_write "$0 must not be executed as privileged user root"
stderr_write
stderr_write "Unlike the actual scanner the sync routine does not need privileges."
stderr_write "Accidental execution as root would prevent later overwriting of"
stderr_write "files with a non-privileged user."
log_err "Denied to run as root"
exit 1
fi
# Always try to get the information when started.
# This also ensures variables like FEED_PRESENT are set.
get_feed_info
# Determine whether a GSF access key is present. If yes,
# then use the Greenbone Security Feed. Else use the
# Greenbone Community Feed.
if [ -e $ACCESS_KEY ]
then
RESTRICTED=1
else
RESTRICTED=0
if [ -z "$COMMUNITY_NVT_RSYNC_FEED" ]; then
COMMUNITY_NVT_RSYNC_FEED=rsync://feed.community.greenbone.net:/nvt-feed
# An alternative syntax which might work if the above doesn't:
# COMMUNITY_NVT_RSYNC_FEED=rsync@feed.community.greenbone.net::/nvt-feed
fi
fi
RSYNC=`command -v rsync`
if [ -z "$TMPDIR" ]; then
SYNC_TMP_DIR=/tmp
# If we have mktemp, create a temporary dir (safer)
if [ -n "`which mktemp`" ]; then
SYNC_TMP_DIR=`mktemp -t -d greenbone-nvt-sync.XXXXXXXXXX` || { echo "ERROR: Cannot create temporary directory for file download" >&2; exit 1 ; }
trap "rm -rf $SYNC_TMP_DIR" EXIT HUP INT TRAP TERM
fi
else
SYNC_TMP_DIR="$TMPDIR"
fi
# Initialize this indicator variable with default assuming the
# feed is not up-to-date.
FEED_CURRENT=0
# This function uses gos-state-manager to get information about the settings.
# If gos-state-manager is not installed the values of the settings can not be
# retrieved.
#
# Input: option
# Output: value as string or empty String if gos-state-manager is not installed
# or option not set
get_value ()
{
value=""
key=$1
if which gos-state-manager 1>/dev/null 2>&1
then
if gos-state-manager get "$key.value" 1>/dev/null 2>&1
then
value="$(gos-state-manager get "$key.value")"
fi
fi
echo "$value"
}
# Creates a restricted access copy of the access key if necessary.
setup_temp_access_key () {
if [ -e "$ACCESS_KEY" ]
then
FILE_ACCESS=`stat -c%a "$ACCESS_KEY" | cut -c2-`
fi
if [ -n "$FILE_ACCESS" ] && [ "00" != "$FILE_ACCESS" ]
then
TEMP_ACCESS_KEY_DIR=`mktemp -d`
TEMP_ACCESS_KEY="$TEMP_ACCESS_KEY_DIR/gsf-access-key"
cp "$ACCESS_KEY" "$TEMP_ACCESS_KEY"
chmod 400 "$TEMP_ACCESS_KEY"
else
TEMP_ACCESS_KEY_DIR=""
TEMP_ACCESS_KEY="$ACCESS_KEY"
fi
}
# Deletes the read-only copy of the access key.
cleanup_temp_access_key () {
if [ -n "$TEMP_ACCESS_KEY_DIR" ]
then
rm -rf "$TEMP_ACCESS_KEY_DIR"
fi
TEMP_ACCESS_KEY_DIR=""
TEMP_ACCESS_KEY=""
}
is_feed_current () {
if [ -z "$FEED_VERSION" ]
then
log_write "Could not determine feed version."
FEED_CURRENT=0
return $FEED_CURRENT
fi
if [ -z "$RSYNC" ]
then
log_notice "rsync not available, skipping feed version test"
FEED_CURRENT=0
rm -rf $FEED_INFO_TEMP_DIR
cleanup_temp_access_key
return 0
fi
FEED_INFO_TEMP_DIR=`mktemp -d`
if [ -e $ACCESS_KEY ]
then
gsmproxy=$(get_value proxy_feed | sed -r -e 's/^.*\/\///' -e 's/:([0-9]+)$/ \1/')
syncport=$(get_value syncport)
if [ "$syncport" ]
then
PORT="$syncport"
fi
read feeduser < $ACCESS_KEY
custid=`awk -F@ 'NR > 1 { exit }; { print $1 }' $ACCESS_KEY`
if [ -z "$feeduser" ] || [ -z "$custid" ]
then
log_err "Could not determine credentials, aborting synchronization."
exit 1
fi
setup_temp_access_key
if [ "$gsmproxy" = "proxy_feed" ] || [ -z "$gsmproxy" ]
then
RSYNC_SSH_PROXY_CMD=""
else
if [ -e $OPENVAS_SYSCONF_DIR/proxyauth ] && [ -r $OPENVAS_SYSCONF_DIR/proxyauth ]
then
RSYNC_SSH_PROXY_CMD="-o \"ProxyCommand corkscrew $gsmproxy %h %p $OPENVAS_SYSCONF_DIR/proxyauth\""
else
RSYNC_SSH_PROXY_CMD="-o \"ProxyCommand corkscrew $gsmproxy %h %p\""
fi
fi
rsync -e "ssh $RSYNC_SSH_OPTS $RSYNC_SSH_PROXY_CMD -p $PORT -i $TEMP_ACCESS_KEY" $RSYNC_OPTIONS $RSYNC_DELETE $RSYNC_COMPRESS $RSYNC_CHMOD "$feeduser"plugin_feed_info.inc $FEED_INFO_TEMP_DIR
#Insert arbitrary sleep here
sleep 5
if [ $? -ne 0 ]
then
log_err "Error: rsync failed."
rm -rf "$FEED_INFO_TEMP_DIR"
exit 1
fi
else
log_notice "No Greenbone Security Feed access key found, falling back to Greenbone Community Feed"
eval "$RSYNC -ltvrP \"$COMMUNITY_NVT_RSYNC_FEED/plugin_feed_info.inc\" \"$FEED_INFO_TEMP_DIR\""
#Insert arbitrary sleep here
sleep 5
if [ $? -ne 0 ]
then
log_err "rsync failed, aborting synchronization."
rm -rf "$FEED_INFO_TEMP_DIR"
exit 1
fi
fi
FEED_VERSION_SERVER=`grep PLUGIN_SET $FEED_INFO_TEMP_DIR/plugin_feed_info.inc | sed -e 's/[^0-9]//g'`
if [ -z "$FEED_VERSION_SERVER" ]
then
log_err "Could not determine server feed version."
rm -rf $FEED_INFO_TEMP_DIR
cleanup_temp_access_key
exit 1
fi
# Check against FEED_VERSION
if [ $FEED_VERSION -lt $FEED_VERSION_SERVER ] ; then
FEED_CURRENT=0
else
FEED_CURRENT=1
fi
# Cleanup
rm -rf "$FEED_INFO_TEMP_DIR"
cleanup_temp_access_key
return $FEED_CURRENT
}
do_rsync_community_feed () {
log_notice "Configured NVT rsync feed: $COMMUNITY_NVT_RSYNC_FEED"
mkdir -p "$NVT_DIR"
eval "$RSYNC -ltvrP $RSYNC_DELETE \"$COMMUNITY_NVT_RSYNC_FEED\" \"$NVT_DIR\" --exclude=plugin_feed_info.inc"
#Insert arbitrary sleep here
sleep 5
if [ $? -ne 0 ] ; then
log_err "rsync failed."
exit 1
fi
eval "$RSYNC -ltvrP $RSYNC_DELETE \"$COMMUNITY_NVT_RSYNC_FEED/plugin_feed_info.inc\" \"$NVT_DIR\""
#Insert arbitrary sleep here
sleep 5
if [ $? -ne 0 ] ; then
log_err "rsync failed."
exit 1
fi
}
sync_nvts(){
if [ $ENABLED -ne 1 ]
then
log_write "NVT synchronization is disabled, exiting."
exit 0
fi
if [ -e $ACCESS_KEY ]
then
log_write "Synchronizing NVTs from the Greenbone Security Feed into $NVT_DIR..."
if [ $FEED_PRESENT -eq 1 ] ; then
FEEDCOUNT=`grep -E "nasl$|inc$" $NVT_DIR/md5sums | wc -l`
log_write "Current status: Using $FEED_NAME at version $FEED_VERSION ($FEEDCOUNT NVTs)"
else
log_write "Current status: No feed installed."
fi
notsynced=1
retried=0
mkdir -p "$NVT_DIR"
read feeduser < $ACCESS_KEY
custid=`awk -F@ 'NR > 1 { exit }; { print $1 }' $ACCESS_KEY`
if [ -z "$feeduser" ] || [ -z "$custid" ]
then
log_err "Could not determine credentials, aborting synchronization."
exit 1
fi
setup_temp_access_key
while [ $notsynced -eq 1 ]
do
gsmproxy=$(get_value proxy_feed | sed -r -e 's/^.*\/\///' -e 's/:([0-9]+)$/ \1/')
syncport=$(get_value syncport)
if [ "$syncport" ]
then
PORT="$syncport"
fi
if [ "$gsmproxy" = "proxy_feed" ] || [ -z "$gsmproxy" ]
then
RSYNC_SSH_PROXY_CMD=""
else
if [ -e $OPENVAS_SYSCONF_DIR/proxyauth ] && [ -r $OPENVAS_SYSCONF_DIR/proxyauth ]; then
RSYNC_SSH_PROXY_CMD="-o \"ProxyCommand corkscrew $gsmproxy %h %p $OPENVAS_SYSCONF_DIR/proxyauth\""
else
RSYNC_SSH_PROXY_CMD="-o \"ProxyCommand corkscrew $gsmproxy %h %p\""
fi
fi
rsync -e "ssh $RSYNC_SSH_OPTS $RSYNC_SSH_PROXY_CMD -p $PORT -i $TEMP_ACCESS_KEY" --exclude=plugin_feed_info.inc $RSYNC_OPTIONS $RSYNC_DELETE $RSYNC_COMPRESS $RSYNC_CHMOD $feeduser $NVT_DIR
#Insert arbitrary sleep here
sleep 5
if [ $? -ne 0 ] ; then
log_err "rsync failed, aborting synchronization."
exit 1
fi
rsync -e "ssh $RSYNC_SSH_OPTS $RSYNC_SSH_PROXY_CMD -p $PORT -i $TEMP_ACCESS_KEY" $RSYNC_OPTIONS $RSYNC_DELETE $RSYNC_COMPRESS $RSYNC_CHMOD "$feeduser"plugin_feed_info.inc $NVT_DIR
#Insert arbitrary sleep here
sleep 5
if [ $? -ne 0 ] ; then
log_err "rsync failed, aborting synchronization."
exit 1
fi
eval "cd \"$NVT_DIR\" ; md5sum -c --status \"$NVT_DIR/md5sums\""
if [ $? -ne 0 ] ; then
if [ -n "$retried" ]
then
log_err "Feed integrity check failed twice, aborting synchronization."
cleanup_temp_access_key
exit 1
else
log_write "The feed integrity check failed. This may be due to a concurrent feed update or other temporary issues."
log_write "Sleeping 15 seconds before retrying ..."
sleep 15
retried=1
fi
else
notsynced=0
fi
done
cleanup_temp_access_key
log_write "Synchronization with the Greenbone Security Feed successful."
get_feed_info
if [ $FEED_PRESENT -eq 1 ] ; then
FEEDCOUNT=`grep -E "nasl$|inc$" $NVT_DIR/md5sums | wc -l`
log_write "Current status: Using $FEED_NAME at version $FEED_VERSION ($FEEDCOUNT NVTs)"
else
log_write "Current status: No feed installed."
fi
else
log_notice "No Greenbone Security Feed access key found, falling back to Greenbone Community Feed"
do_rsync_community_feed
fi
}
do_self_test ()
{
MD5SUM_AVAIL=`command -v md5sum`
if [ $? -ne 0 ] ; then
SELFTEST_FAIL=1
stderr_write "The md5sum binary could not be found."
fi
RSYNC_AVAIL=`command -v rsync`
if [ $? -ne 0 ] ; then
SELFTEST_FAIL=1
stderr_write "The rsync binary could not be found."
fi
}
do_describe ()
{
echo "This script synchronizes an NVT collection with the '$FEED_NAME'."
echo "The '$FEED_NAME' is provided by '$FEED_VENDOR'."
echo "Online information about this feed: '$FEED_HOME'."
}
do_feedversion () {
if [ $FEED_PRESENT -eq 1 ] ; then
echo $FEED_VERSION
else
stderr_write "The file containing the feed version could not be found."
exit 1
fi
}
do_sync ()
{
do_self_test
if [ $SELFTEST_FAIL -ne 0 ] ; then
exit $SELFTEST_FAIL
fi
if [ $FEED_CURRENT -eq 1 ]
then
log_write "Feed is already current, skipping synchronization."
else
(
flock -n 9
if [ $? -eq 1 ] ; then
log_warning "Another process related to the feed update is already running"
exit 1
fi
date > $OPENVAS_RUN_DIR/feed-update.lock
sync_nvts
echo -n $OPENVAS_RUN_DIR/feed-update.lock
)9>$OPENVAS_RUN_DIR/feed-update.lock
fi
}
do_help () {
echo "$0: Sync NVT data"
echo " --describe display current feed info"
echo " --feedcurrent just check if feed is up-to-date"
echo " --feedversion display version of this feed"
echo " --help display this help"
echo " --identify display information"
echo " --nvtdir dir set dir as NVT directory"
echo " --selftest perform self-test and set exit code"
echo " --verbose makes the sync process print details"
echo " --version display version"
echo ""
echo ""
echo "Environment variables:"
echo "NVT_DIR where to extract plugins (absolute path)"
echo "PRIVATE_SUBDIR subdirectory of \$NVT_DIR to exclude from synchronization"
echo "TMPDIR temporary directory used to download the files"
echo "Note that you can use standard ones as well (e.g. RSYNC_PROXY) for rsync"
echo ""
exit 0
}
while test $# -gt 0; do
case "$1" in
--version)
echo $VERSION
exit 0
;;
--identify)
echo "NVTSYNC|$SCRIPT_NAME|$VERSION|$FEED_NAME|$RESTRICTED|NVTSYNC"
exit 0
;;
--selftest)
do_self_test
exit $SELFTEST_FAIL
;;
--describe)
do_describe
exit 0
;;
--feedversion)
do_feedversion
exit 0
;;
--help)
do_help
exit 0
;;
--nvt-dir)
NVT_DIR="$2"
shift
;;
--feedcurrent)
is_feed_current
exit $?
;;
--verbose)
RSYNC_VERBOSE="-v"
;;
esac
shift
done
do_sync
exit 0
Your should fix your firewall and timeouts and not the script
@Lukas - oh yeah, no doubt. I do not disagree. This was purely to help figure out what the problem was. I’ll look at NAT/firewall to see what I can fix there. Here at the “home office” it’s a consumer grade firewall, and even with custom firmware I’m not certain that’s an option.
Thanks for sharing this!
Having seen your firewall rules here I understand the issue better. I made a couple of changes to my NAT gateway, to no avail. But at least I have an understanding of what’s happening.
For the rest of the audience, the firewall rule
REJECT tcp – * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:873 flags:0x17/0x02 #conn src/32 > 1 reject-with tcp-reset
disallows any more than one connection per source address at a time. There are no timers or anything like that. So if you see what we’re seeing, it’s something to do with a connection that’s being held open for just a bit too long.
@dimante - I tried disabling NAT acceleration and it did not fix my issue here. I’m testing with Asuswrt-merlin on my home router, so my options are limited as to what I can tweak. But you might see if there are any hold-open timers or similar options that are causing TCP sessions to stay open.
So NVT feed is gone again and gvmd seems to report loading NVTs then apparently stuck in a loop… Why NVT, why… Other feeds are updated and done… Need to find the root cause of this issue…
Here is the odd part about this… Removing the following files from the run directory 0 byte and nothing more the nvt update finishes and it’s good to go…
gvm-checking
gvm-create-functions
gvm-helping
gvm-migrating
gvm-serving
Not sure why all this manual intervention is required but removing these allowed the process to complete and gvmd to become operational and the NVT feed to reappear…
Thoughts?
So what I explained above works everytime. Now looking in the logs (even though everything is working) I see this repetitively:
md manage:WARNING:2020-06-21 15h12.01 utc:4712: open_secinfo_lockfile: failed to open lock file ‘/tmp/gvm-sync-scap’: No such file or directory
md manage:WARNING:2020-06-21 15h12.01 utc:4713: open_secinfo_lockfile: failed to open lock file ‘/tmp/gvm-sync-cert’: No such file or directory
md manage:WARNING:2020-06-21 15h12.11 utc:4725: open_secinfo_lockfile: failed to open lock file ‘/tmp/gvm-sync-scap’: No such file or directory
md manage:WARNING:2020-06-21 15h12.11 utc:4727: open_secinfo_lockfile: failed to open lock file ‘/tmp/gvm-sync-cert’: No such file or directory
md manage:WARNING:2020-06-21 15h12.21 utc:4740: open_secinfo_lockfile: failed to open lock file ‘/tmp/gvm-sync-scap’: No such file or directory
md manage:WARNING:2020-06-21 15h12.21 utc:4741: open_secinfo_lockfile: failed to open lock file ‘/tmp/gvm-sync-cert’: No such file or directory
md manage:WARNING:2020-06-21 15h12.31 utc:4753: open_secinfo_lockfile: failed to open lock file ‘/tmp/gvm-sync-scap’: No such file or directory
md manage:WARNING:2020-06-21 15h12.31 utc:4756: open_secinfo_lockfile: failed to open lock file ‘/tmp/gvm-sync-cert’: No such file or directory
md manage:WARNING:2020-06-21 15h12.41 utc:4765: open_secinfo_lockfile: failed to open lock file ‘/tmp/gvm-sync-scap’: No such file or directory
md manage:WARNING:2020-06-21 15h12.41 utc:4766: open_secinfo_lockfile: failed to open lock file ‘/tmp/gvm-sync-cert’: No such file or directory
Why is it looking for these files. Where is the setting for this as the path for me would be /opt/gvm/tmp
Thanks!
Adding sleep commands into the script did the trick to me as well. Thanks for sharing this !
Yeah adding sleep commands works for me too. When there’s a whole thread of different people having the same issue, with the same resolution, kind of hard to blame NAT or the firewall for this. Unless every firewall on the planet is broken, seems to me that the common denominator here is the script. Probably just opening the next connection so fast that the mirror doesn’t have time to realize the first connection is closed. Sounds like a mirror problem, not a NAT / firewall issue. I think someone mentioned the repos are on NIST servers (which are incredibly slow). I’m more inclined to believe it’s a problem on that end than on all the users having the same problem’s end with the common denominator of this script and the NIST servers.
Yes, I completely agree with this. I have had the same problem with this script with different firewalls manufacturers. Systematically; adding the sleep command fixed the issue.
So very hard to continue blaming the firewalls / NAT for this… I wonder if the professional feeds have the same problem…
GSF is behind a different system where double connections are allowed and HTTPS port 443 as SSH are possible due to a different authentication and authorization process.