Feed sync via rsync aborts

Please be aware that only one session at the time is possible, if you are not direct connected to the internet it is mostly a other GVM system, or a network device that might block you from syncing. NAT-Gates and firewalls trend to keep sessions alive, even if your side is down :wink:

1 Like

After hours and hours of going through loads and loads of this I have found that all services can be green yet the web interface not be ready, scanner failing verification etc. Here is my write-up regarding my experience with this application in regard to feed updates. Please feel free to expand on this further. This software is really good and I have contributed to many open source projects so a ton of time goes in to building and maintaining it so thank you for providing it!!

Update Process (Paths based on Debian build from source):

Update in this order:

greenbone-nvt-sync
Need to wait for this to complete. If it dies in the middle of the sync a .lock file will be left in the /opt/gvm/var/run/*.lock this lock file will need to be removed and then run the greenbone-nvt-sync command again. Failure to do this and moving on to the next sync will certainly break things!

greenbone-certdata-sync
Follows nvt sync and usually completes without error. Again check for the presence of a .lock file and run the command again until it completes

greenbone-scapdata-sync
Follow the same guidelines as above
When these commands are complete check the gvmd service status

systemctl status gvmd
Output similar to the following will be shown:

â gvmd.service - Job that runs the gvm daemon
Loaded: loaded (/etc/systemd/system/gvmd.service; enabled; vendor preset: ena
Active: active (running) since Mon 2020-06-08 20:15:50 CDT; 47min ago
Docs: man:gvm
Process: 661 ExecStart=/opt/gvm/sbin/gvmd --osp-vt-update=/opt/gvm/var/run/osp
Main PID: 667 (gvmd)
Tasks: 3 (limit: 4689)
Memory: 226.9M
CGroup: /system.slice/gvmd.service
ââ 667 gvmd: Waiting for incoming connections
ââ6060 gvmd: Reloading NVTs
ââ6062 gvmd: OSP: Updating NVT cache

Until the service is done reloading operations it’s best not to try to run scans or disrupt it in any way. The updating takes a bit so when you do this it’s probably best to let it run through the evening then use it in the AM to scan.

A finished output looks similar to the following:

â gvmd.service - Job that runs the gvm daemon
Loaded: loaded (/etc/systemd/system/gvmd.service; enabled; vendor preset: ena
Active: active (running) since Mon 2020-06-08 20:15:50 CDT; 1h 0min ago
Docs: man:gvm
Process: 661 ExecStart=/opt/gvm/sbin/gvmd --osp-vt-update=/opt/gvm/var/run/osp
Main PID: 667 (gvmd)
Tasks: 1 (limit: 4689)
Memory: 224.5M
CGroup: /system.slice/gvmd.service
ââ667 gvmd: Waiting for incoming connections

The following services should also be checked as well for a running state:

systemctl status gsad

systemctl status ospd-openvas

If all three services are running, the system should open and function correctly.

-If the feeds get hosed up the software will not run the web browser and you are rolling back (VM) or restoring from a backup. It’s one of the things you have to deal with. The software is free after all.

-D-

If someone is having good luck with running updates thorough cron I would be interested on how you are handling the update failures when they occur.

@Lukas - thanks for that info. I’m reasonably certain there weren’t other sync processes running from other systems. It’s possible that a NAT gateway was holding open a stale session, but I can test that by simply restarting my gateway and the Greenbone virtual machine.

  • Update: restarted gateway and VM. No joy. However, I think I’m narrowing in on my problem, and perhaps this can help someone else. When I manually run rsync from the command line on my physical machine, I am able to connect and retrieve files with no issue. However, on the Greenbone VM, which is in VirtualBox, I’m having issues. It could be the bridged connection, or something like that causing issues. I’ll test some more and report back.

  • Update 2: the failures seemed to be random in nature, so I was suspecting network-ey things as @Lukas suggested. I added some arbitrary sleep statements into the greenbone-nvt-sync script to give it time in between rsync commands to the feed update server. This seems to work all the time. So, perhaps it is some stale session in a NAT gateway or firewall, but I can’t prove it.

  • Update 3: Arbitrary sleep statements seem to make it work every time. Taking them out immediately caused a failure. Putting them back in, and it works again. So my suspicion is that something is causing it to look like it’s two sessions, when it’s only one script running but with rsync commands really close together in timing. When I spread out the timing of the multiple rsync commands, it works. I hate arbitrary sleep statements, but it works.

How did I get here? I noticed that the first “receiving incremental file list” message succeeded, and subsequent ones in the script failed. Why would one work and another fail in the same script? Probably because there’s a stale session caused by NAT, firewall, or VirtualBox.

Not sure what the issue is yet. From the VM, it’s working right now. But a few minutes ago it wasn’t. No substantial changes.

  • Final update: with arbitrary sleep statements in greenbone-scapdata-sync, greenbone-certdata-sync, and greenbone-nvt-sync after each and every rsync command, it works every time.

It’s a very kludgy workaround and I’m not proud of it. But it gets around the NAT/firewall/VM issues or whatever is causing stale session info on the server side.

Care to share the solution?

Something like this. Arbitrary sleep statements after each call to rsync. It’s definitely not pretty. I did this in all three sync scripts (greenbone-nvt-sync, greenbone-certdata-sync, greenbone-scapdata-sync). Let me know if it works for you as well.

#!/bin/sh
# Copyright (C) 2009-2020 Greenbone Networks GmbH
#
# SPDX-License-Identifier: GPL-2.0-or-later
#
# This program is free software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 2
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA.

# This script updates the local Network Vulnerability Tests (NVTs) from the
# Greenbone Security Feed (GSF) or the Greenbone Community Feed (GCF). 

VERSION=@OPENVAS_VERSION@

# SETTINGS
# ========

# PRIVATE_SUBDIR defines a subdirectory of the NVT directory that is excluded
# from the feed sync. This is where to place your own NVTs.
if [ -z "$PRIVATE_SUBDIR" ]
then
  PRIVATE_SUBDIR="private"
fi

# RSYNC_DELETE controls whether files which are not part of the repository will
# be removed from the local directory after synchronization. The default value
# for this setting is
# "--delete --exclude \"$PRIVATE_SUBDIR/\"",
# which means that files which are not part of the feed or private directory
# will be deleted.
RSYNC_DELETE="--delete --exclude $PRIVATE_SUBDIR/"

# RSYNC_SSH_OPTS contains options which should be passed to ssh for the rsync
# connection to the repository.
RSYNC_SSH_OPTS="-o \"UserKnownHostsFile=/dev/null\" -o \"StrictHostKeyChecking=no\""

# RSYNC_COMPRESS specifies the compression level to use for the rsync connection.
RSYNC_COMPRESS="--compress-level=9"

# RSYNC_CHMOD specifies the permissions to chmod the files to.
RSYNC_CHMOD="--perms --chmod=Fugo+r,Fug+w,Dugo-s,Dugo+rx,Dug+w"

# Verbosity flag for rsync. "-q" means a quiet rsync, "-v" a verbose rsync.
RSYNC_VERBOSE="-q"

# RSYNC_OPTIONS controls the general parameters for the rsync connection.
RSYNC_OPTIONS="--links --times --omit-dir-times $RSYNC_VERBOSE --recursive --partial --progress"

# Script and feed information which will be made available to user through
# command line options and automated tools.
# Script name which will be used for logging
SCRIPT_NAME="greenbone-nvt-sync"

# Result of selftest () is stored here. If it is not 0, the selftest has failed
# and the sync script is unlikely to work.
SELFTEST_FAIL=0

# Port to use for synchronization. Default value is 24.
PORT=24

# Directory where the OpenVAS configuration is located
OPENVAS_SYSCONF_DIR="@OPENVAS_SYSCONF_DIR@"

# Directory where the feed update lock file will be placed.
OPENVAS_RUN_DIR="@OPENVAS_RUN_DIR@"

# Location of the GSF Access Key
ACCESS_KEY="@GVM_ACCESS_KEY_DIR@/gsf-access-key"

# If ENABLED is set to 0, the sync script will not perform a synchronization.
ENABLED=1

# LOG_CMD defines the command to use for logging. To have logger log to stderr
# as well as syslog, add "-s" here. The logging facility is checked. In case of error
# all will be logged in the standard error and the socket error check will be
# disabled.
LOG_CMD="logger -t $SCRIPT_NAME"

check_logger () {
  logger --socket-error=on -p daemon.info -t $SCRIPT_NAME "Checking logger" --no-act 1>/dev/null 2>&1
  if [ $? -gt 0 ]
  then
    LOG_CMD="logger --socket-error=off -s -t $SCRIPT_NAME"
    $LOG_CMD -p daemon.warning "The log facility is not working as expected. All messages will be written to the standard error stream."
  fi
}
check_logger


# Source configuration file if it is readable
[ -r $OPENVAS_SYSCONF_DIR/greenbone-nvt-sync.conf ] && . $OPENVAS_SYSCONF_DIR/greenbone-nvt-sync.conf

# NVT_DIR is the place where the NVTs are located.
if [ -z "$NVT_DIR" ]
then
  NVT_DIR="@OPENVAS_NVT_DIR@"
fi

log_write () {
  $LOG_CMD -p daemon.notice $1
}

log_debug () {
  $LOG_CMD -p daemon.debug "$1"
}

log_info () {
  $LOG_CMD -p daemon.info "$1"
}

log_notice () {
  $LOG_CMD -p daemon.notice "$1"
}

log_warning () {
  $LOG_CMD -p daemon.warning "$1"
}

log_err () {
  $LOG_CMD -p daemon.err "$1"
}

stderr_write ()
{
  echo "$1" > /dev/stderr
}

# Read the general information about the feed origin from
# the file "plugin_feed_info.inc" inside the feed directory.
get_feed_info ()
{
  INFOFILE="$NVT_DIR/plugin_feed_info.inc"
  if [ -r $INFOFILE ] ; then
    FEED_VERSION=`grep PLUGIN_SET $INFOFILE | sed -e 's/[^0-9]//g'`
    FEED_NAME=`awk -F\" '/PLUGIN_FEED/ { print $2 }' $INFOFILE`
    FEED_VENDOR=`awk -F\" '/FEED_VENDOR/ { print $2 }' $INFOFILE`
    FEED_HOME=`awk -F\" '/FEED_HOME/ { print $2 }' $INFOFILE`
    FEED_PRESENT=1
  else
    FEED_PRESENT=0
  fi

  if [ -z "$FEED_NAME" ] ; then
    FEED_NAME="Unidentified Feed"
  fi

  if [ -z "$FEED_VENDOR" ] ; then
    FEED_VENDOR="Unidentified Vendor"
  fi

  if [ -z "$FEED_HOME" ] ; then
    FEED_HOME="Unidentified Feed Homepage"
  fi
}

# Prevent that root executes this script
if [ "`id -u`" -eq "0" ]
then
  stderr_write "$0 must not be executed as privileged user root"
  stderr_write
  stderr_write "Unlike the actual scanner the sync routine does not need privileges."
  stderr_write "Accidental execution as root would prevent later overwriting of"
  stderr_write "files with a non-privileged user."

  log_err "Denied to run as root"
  exit 1
fi

# Always try to get the information when started.
# This also ensures variables like FEED_PRESENT are set.
get_feed_info

# Determine whether a GSF access key is present. If yes,
# then use the Greenbone Security Feed. Else use the
# Greenbone Community Feed.
if [ -e $ACCESS_KEY ]
then
  RESTRICTED=1
else
  RESTRICTED=0

  if [ -z "$COMMUNITY_NVT_RSYNC_FEED" ]; then
    COMMUNITY_NVT_RSYNC_FEED=rsync://feed.community.greenbone.net:/nvt-feed
    # An alternative syntax which might work if the above doesn't:
    # COMMUNITY_NVT_RSYNC_FEED=rsync@feed.community.greenbone.net::/nvt-feed
  fi
fi

RSYNC=`command -v rsync`

if [ -z "$TMPDIR" ]; then
  SYNC_TMP_DIR=/tmp
  # If we have mktemp, create a temporary dir (safer)
  if [ -n "`which mktemp`" ]; then
    SYNC_TMP_DIR=`mktemp -t -d greenbone-nvt-sync.XXXXXXXXXX` || { echo "ERROR: Cannot create temporary directory for file download" >&2; exit 1 ; }
    trap "rm -rf $SYNC_TMP_DIR" EXIT HUP INT TRAP TERM
  fi
else
  SYNC_TMP_DIR="$TMPDIR"
fi

# Initialize this indicator variable with default assuming the
# feed is not up-to-date.
FEED_CURRENT=0

# This function uses gos-state-manager to get information about the settings.
# If gos-state-manager is not installed the values of the settings can not be
# retrieved.
#
# Input: option
# Output: value as string or empty String if gos-state-manager is not installed
#         or option not set
get_value ()
{
  value=""
  key=$1
  if which gos-state-manager 1>/dev/null 2>&1
  then
    if gos-state-manager get "$key.value" 1>/dev/null 2>&1
    then
      value="$(gos-state-manager get "$key.value")"
    fi
  fi
  echo "$value"
}

# Creates a restricted access copy of the access key if necessary.
setup_temp_access_key () {
  if [ -e "$ACCESS_KEY" ]
  then
    FILE_ACCESS=`stat -c%a "$ACCESS_KEY" | cut -c2-`
  fi
  if [ -n "$FILE_ACCESS" ] && [ "00" != "$FILE_ACCESS" ]
  then
    TEMP_ACCESS_KEY_DIR=`mktemp -d`
    TEMP_ACCESS_KEY="$TEMP_ACCESS_KEY_DIR/gsf-access-key"
    cp "$ACCESS_KEY" "$TEMP_ACCESS_KEY"
    chmod 400 "$TEMP_ACCESS_KEY"
  else
    TEMP_ACCESS_KEY_DIR=""
    TEMP_ACCESS_KEY="$ACCESS_KEY"
  fi
}

# Deletes the read-only copy of the access key.
cleanup_temp_access_key () {
  if [ -n "$TEMP_ACCESS_KEY_DIR" ]
  then
    rm -rf "$TEMP_ACCESS_KEY_DIR"
  fi
  TEMP_ACCESS_KEY_DIR=""
  TEMP_ACCESS_KEY=""
}

is_feed_current () {
  if [ -z "$FEED_VERSION" ]
  then
    log_write "Could not determine feed version."
    FEED_CURRENT=0
    return $FEED_CURRENT
  fi

  if [ -z "$RSYNC" ]
  then
    log_notice "rsync not available, skipping feed version test"
    FEED_CURRENT=0
    rm -rf $FEED_INFO_TEMP_DIR
    cleanup_temp_access_key
    return 0
  fi

  FEED_INFO_TEMP_DIR=`mktemp -d`

  if [ -e $ACCESS_KEY ]
  then
    gsmproxy=$(get_value proxy_feed | sed -r -e 's/^.*\/\///' -e 's/:([0-9]+)$/ \1/')
    syncport=$(get_value syncport)
    if [ "$syncport" ]
    then
      PORT="$syncport"
    fi

    read feeduser < $ACCESS_KEY
    custid=`awk -F@ 'NR > 1 { exit }; { print $1 }' $ACCESS_KEY`
    if [ -z "$feeduser" ] || [ -z "$custid" ]
    then
      log_err "Could not determine credentials, aborting synchronization."
      exit 1
    fi

    setup_temp_access_key

    if [ "$gsmproxy" = "proxy_feed" ] || [ -z "$gsmproxy" ]
    then
      RSYNC_SSH_PROXY_CMD=""
    else
      if [ -e $OPENVAS_SYSCONF_DIR/proxyauth ] && [ -r $OPENVAS_SYSCONF_DIR/proxyauth ]
      then
        RSYNC_SSH_PROXY_CMD="-o \"ProxyCommand corkscrew $gsmproxy %h %p $OPENVAS_SYSCONF_DIR/proxyauth\""
      else
        RSYNC_SSH_PROXY_CMD="-o \"ProxyCommand corkscrew $gsmproxy %h %p\""
      fi
    fi

    rsync -e "ssh $RSYNC_SSH_OPTS $RSYNC_SSH_PROXY_CMD -p $PORT -i $TEMP_ACCESS_KEY" $RSYNC_OPTIONS $RSYNC_DELETE $RSYNC_COMPRESS $RSYNC_CHMOD "$feeduser"plugin_feed_info.inc $FEED_INFO_TEMP_DIR

    #Insert arbitrary sleep here
    sleep 5

    if [ $? -ne 0 ]
    then
      log_err "Error: rsync failed."
      rm -rf "$FEED_INFO_TEMP_DIR"
      exit 1
    fi
  else
    log_notice "No Greenbone Security Feed access key found, falling back to Greenbone Community Feed"
    eval "$RSYNC -ltvrP \"$COMMUNITY_NVT_RSYNC_FEED/plugin_feed_info.inc\" \"$FEED_INFO_TEMP_DIR\""

    #Insert arbitrary sleep here
    sleep 5

    if [ $? -ne 0 ]
    then
      log_err "rsync failed, aborting synchronization."
      rm -rf "$FEED_INFO_TEMP_DIR"
      exit 1
    fi
  fi

  FEED_VERSION_SERVER=`grep PLUGIN_SET $FEED_INFO_TEMP_DIR/plugin_feed_info.inc | sed -e 's/[^0-9]//g'`

  if [ -z "$FEED_VERSION_SERVER" ]
  then
    log_err "Could not determine server feed version."
    rm -rf $FEED_INFO_TEMP_DIR
    cleanup_temp_access_key
    exit 1
  fi
  # Check against FEED_VERSION
  if [ $FEED_VERSION -lt $FEED_VERSION_SERVER ] ; then
    FEED_CURRENT=0
  else
    FEED_CURRENT=1
  fi
  # Cleanup
  rm -rf "$FEED_INFO_TEMP_DIR"
  cleanup_temp_access_key

  return $FEED_CURRENT
}

do_rsync_community_feed () {
  log_notice "Configured NVT rsync feed: $COMMUNITY_NVT_RSYNC_FEED"
  mkdir -p "$NVT_DIR"
  eval "$RSYNC -ltvrP $RSYNC_DELETE \"$COMMUNITY_NVT_RSYNC_FEED\" \"$NVT_DIR\" --exclude=plugin_feed_info.inc"

  #Insert arbitrary sleep here
  sleep 5

  if [ $? -ne 0 ] ; then
    log_err "rsync failed."
    exit 1
  fi
  eval "$RSYNC -ltvrP $RSYNC_DELETE \"$COMMUNITY_NVT_RSYNC_FEED/plugin_feed_info.inc\" \"$NVT_DIR\""

  #Insert arbitrary sleep here
  sleep 5

  if [ $? -ne 0 ] ; then
    log_err "rsync failed."
    exit 1
  fi
}

sync_nvts(){
  if [ $ENABLED -ne 1 ]
  then
    log_write "NVT synchronization is disabled, exiting."
    exit 0
  fi

  if [ -e $ACCESS_KEY ]
  then
    log_write "Synchronizing NVTs from the Greenbone Security Feed into $NVT_DIR..."
    if [ $FEED_PRESENT -eq 1 ] ; then
      FEEDCOUNT=`grep -E "nasl$|inc$" $NVT_DIR/md5sums | wc -l`
      log_write "Current status: Using $FEED_NAME at version $FEED_VERSION ($FEEDCOUNT NVTs)"
    else
      log_write "Current status: No feed installed."
    fi
    notsynced=1
    retried=0

    mkdir -p "$NVT_DIR"
    read feeduser < $ACCESS_KEY
    custid=`awk -F@ 'NR > 1 { exit }; { print $1 }' $ACCESS_KEY`
    if [ -z "$feeduser" ] || [ -z "$custid" ]
    then
      log_err "Could not determine credentials, aborting synchronization."
      exit 1
    fi

    setup_temp_access_key

    while [ $notsynced -eq 1 ]
    do

      gsmproxy=$(get_value proxy_feed | sed -r -e 's/^.*\/\///' -e 's/:([0-9]+)$/ \1/')
      syncport=$(get_value syncport)
      if [ "$syncport" ]
      then
        PORT="$syncport"
      fi

      if [ "$gsmproxy" = "proxy_feed" ] || [ -z "$gsmproxy" ]
      then
        RSYNC_SSH_PROXY_CMD=""
      else
        if [ -e $OPENVAS_SYSCONF_DIR/proxyauth ] && [ -r $OPENVAS_SYSCONF_DIR/proxyauth ]; then
          RSYNC_SSH_PROXY_CMD="-o \"ProxyCommand corkscrew $gsmproxy %h %p $OPENVAS_SYSCONF_DIR/proxyauth\""
        else
          RSYNC_SSH_PROXY_CMD="-o \"ProxyCommand corkscrew $gsmproxy %h %p\""
        fi
      fi
      rsync -e "ssh $RSYNC_SSH_OPTS $RSYNC_SSH_PROXY_CMD -p $PORT -i $TEMP_ACCESS_KEY" --exclude=plugin_feed_info.inc $RSYNC_OPTIONS $RSYNC_DELETE $RSYNC_COMPRESS $RSYNC_CHMOD $feeduser $NVT_DIR

      #Insert arbitrary sleep here
      sleep 5

      if [ $? -ne 0 ]  ; then
        log_err "rsync failed, aborting synchronization."
        exit 1
      fi
      rsync -e "ssh $RSYNC_SSH_OPTS $RSYNC_SSH_PROXY_CMD -p $PORT -i $TEMP_ACCESS_KEY" $RSYNC_OPTIONS $RSYNC_DELETE $RSYNC_COMPRESS $RSYNC_CHMOD "$feeduser"plugin_feed_info.inc $NVT_DIR

      #Insert arbitrary sleep here
      sleep 5

      if [ $? -ne 0 ]  ; then
        log_err "rsync failed, aborting synchronization."
        exit 1
      fi
      eval "cd \"$NVT_DIR\" ; md5sum -c --status \"$NVT_DIR/md5sums\""
      if [ $? -ne 0 ]  ; then
        if [ -n "$retried" ]
        then
          log_err "Feed integrity check failed twice, aborting synchronization."
          cleanup_temp_access_key
          exit 1
        else
          log_write "The feed integrity check failed. This may be due to a concurrent feed update or other temporary issues."
          log_write "Sleeping 15 seconds before retrying ..."
          sleep 15
          retried=1
        fi
      else
        notsynced=0
      fi
    done
    cleanup_temp_access_key
    log_write "Synchronization with the Greenbone Security Feed successful."
    get_feed_info
    if [ $FEED_PRESENT -eq 1 ] ; then
      FEEDCOUNT=`grep -E "nasl$|inc$" $NVT_DIR/md5sums | wc -l`
      log_write "Current status: Using $FEED_NAME at version $FEED_VERSION ($FEEDCOUNT NVTs)"
    else
      log_write "Current status: No feed installed."
    fi
  else
    log_notice "No Greenbone Security Feed access key found, falling back to Greenbone Community Feed"
    do_rsync_community_feed
  fi
}

do_self_test ()
{
  MD5SUM_AVAIL=`command -v md5sum`
  if [ $? -ne 0 ] ; then
    SELFTEST_FAIL=1
    stderr_write "The md5sum binary could not be found."
  fi

  RSYNC_AVAIL=`command -v rsync`
  if [ $? -ne 0 ] ; then
    SELFTEST_FAIL=1
    stderr_write "The rsync binary could not be found."
  fi
}

do_describe ()
{
  echo "This script synchronizes an NVT collection with the '$FEED_NAME'."
  echo "The '$FEED_NAME' is provided by '$FEED_VENDOR'."
  echo "Online information about this feed: '$FEED_HOME'."
}

do_feedversion () {
  if [ $FEED_PRESENT -eq 1 ] ; then
    echo $FEED_VERSION
  else
    stderr_write "The file containing the feed version could not be found."
    exit 1
  fi
}

do_sync ()
{
  do_self_test
  if [ $SELFTEST_FAIL -ne 0 ] ; then
    exit $SELFTEST_FAIL
  fi

  if [ $FEED_CURRENT -eq 1 ]
  then
    log_write "Feed is already current, skipping synchronization."
  else
    (
      flock -n 9
      if [ $? -eq 1 ] ; then
          log_warning "Another process related to the feed update is already running"
          exit 1
      fi
      date > $OPENVAS_RUN_DIR/feed-update.lock
      sync_nvts
      echo -n $OPENVAS_RUN_DIR/feed-update.lock
    )9>$OPENVAS_RUN_DIR/feed-update.lock
  fi
}

do_help () {
  echo "$0: Sync NVT data"
  echo " --describe     display current feed info"
  echo " --feedcurrent  just check if feed is up-to-date"
  echo " --feedversion  display version of this feed"
  echo " --help         display this help"
  echo " --identify     display information"
  echo " --nvtdir dir   set dir as NVT directory"
  echo " --selftest     perform self-test and set exit code"
  echo " --verbose      makes the sync process print details"
  echo " --version      display version"
  echo ""
  echo ""
  echo "Environment variables:"
  echo "NVT_DIR         where to extract plugins (absolute path)"
  echo "PRIVATE_SUBDIR  subdirectory of \$NVT_DIR to exclude from synchronization"
  echo "TMPDIR          temporary directory used to download the files"
  echo "Note that you can use standard ones as well (e.g. RSYNC_PROXY) for rsync"
  echo ""
  exit 0
}

while test $# -gt 0; do
  case "$1" in
    --version)
      echo $VERSION
      exit 0
      ;;
    --identify)
      echo "NVTSYNC|$SCRIPT_NAME|$VERSION|$FEED_NAME|$RESTRICTED|NVTSYNC"
      exit 0
      ;;
    --selftest)
      do_self_test
      exit $SELFTEST_FAIL
      ;;
    --describe)
      do_describe
      exit 0
      ;;
    --feedversion)
      do_feedversion
      exit 0
      ;;
    --help)
      do_help
      exit 0
      ;;
    --nvt-dir)
      NVT_DIR="$2"
      shift
      ;;
    --feedcurrent)
      is_feed_current
      exit $?
      ;;
    --verbose)
      RSYNC_VERBOSE="-v"
      ;;
  esac
  shift
done

do_sync

exit 0

Your should fix your firewall and timeouts and not the script :wink:

@Lukas - oh yeah, no doubt. I do not disagree. This was purely to help figure out what the problem was. I’ll look at NAT/firewall to see what I can fix there. Here at the “home office” it’s a consumer grade firewall, and even with custom firmware I’m not certain that’s an option.

Thanks for sharing this!

Having seen your firewall rules here I understand the issue better. I made a couple of changes to my NAT gateway, to no avail. But at least I have an understanding of what’s happening.

For the rest of the audience, the firewall rule

REJECT tcp – * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:873 flags:0x17/0x02 #conn src/32 > 1 reject-with tcp-reset

disallows any more than one connection per source address at a time. There are no timers or anything like that. So if you see what we’re seeing, it’s something to do with a connection that’s being held open for just a bit too long.

@dimante - I tried disabling NAT acceleration and it did not fix my issue here. I’m testing with Asuswrt-merlin on my home router, so my options are limited as to what I can tweak. But you might see if there are any hold-open timers or similar options that are causing TCP sessions to stay open.

So NVT feed is gone again and gvmd seems to report loading NVTs then apparently stuck in a loop… Why NVT, why… Other feeds are updated and done… Need to find the root cause of this issue…

Here is the odd part about this… Removing the following files from the run directory 0 byte and nothing more the nvt update finishes and it’s good to go…

gvm-checking
gvm-create-functions
gvm-helping
gvm-migrating
gvm-serving

Not sure why all this manual intervention is required but removing these allowed the process to complete and gvmd to become operational and the NVT feed to reappear…

Thoughts?

So what I explained above works everytime. Now looking in the logs (even though everything is working) I see this repetitively:

md manage:WARNING:2020-06-21 15h12.01 utc:4712: open_secinfo_lockfile: failed to open lock file ‘/tmp/gvm-sync-scap’: No such file or directory
md manage:WARNING:2020-06-21 15h12.01 utc:4713: open_secinfo_lockfile: failed to open lock file ‘/tmp/gvm-sync-cert’: No such file or directory
md manage:WARNING:2020-06-21 15h12.11 utc:4725: open_secinfo_lockfile: failed to open lock file ‘/tmp/gvm-sync-scap’: No such file or directory
md manage:WARNING:2020-06-21 15h12.11 utc:4727: open_secinfo_lockfile: failed to open lock file ‘/tmp/gvm-sync-cert’: No such file or directory
md manage:WARNING:2020-06-21 15h12.21 utc:4740: open_secinfo_lockfile: failed to open lock file ‘/tmp/gvm-sync-scap’: No such file or directory
md manage:WARNING:2020-06-21 15h12.21 utc:4741: open_secinfo_lockfile: failed to open lock file ‘/tmp/gvm-sync-cert’: No such file or directory
md manage:WARNING:2020-06-21 15h12.31 utc:4753: open_secinfo_lockfile: failed to open lock file ‘/tmp/gvm-sync-scap’: No such file or directory
md manage:WARNING:2020-06-21 15h12.31 utc:4756: open_secinfo_lockfile: failed to open lock file ‘/tmp/gvm-sync-cert’: No such file or directory
md manage:WARNING:2020-06-21 15h12.41 utc:4765: open_secinfo_lockfile: failed to open lock file ‘/tmp/gvm-sync-scap’: No such file or directory
md manage:WARNING:2020-06-21 15h12.41 utc:4766: open_secinfo_lockfile: failed to open lock file ‘/tmp/gvm-sync-cert’: No such file or directory
Why is it looking for these files. Where is the setting for this as the path for me would be /opt/gvm/tmp

Thanks!

1 Like

Adding sleep commands into the script did the trick to me as well. Thanks for sharing this !

Yeah adding sleep commands works for me too. When there’s a whole thread of different people having the same issue, with the same resolution, kind of hard to blame NAT or the firewall for this. Unless every firewall on the planet is broken, seems to me that the common denominator here is the script. Probably just opening the next connection so fast that the mirror doesn’t have time to realize the first connection is closed. Sounds like a mirror problem, not a NAT / firewall issue. I think someone mentioned the repos are on NIST servers (which are incredibly slow). I’m more inclined to believe it’s a problem on that end than on all the users having the same problem’s end with the common denominator of this script and the NIST servers.

Yes, I completely agree with this. I have had the same problem with this script with different firewalls manufacturers. Systematically; adding the sleep command fixed the issue.

So very hard to continue blaming the firewalls / NAT for this… I wonder if the professional feeds have the same problem…

GSF is behind a different system where double connections are allowed and HTTPS port 443 as SSH are possible due to a different authentication and authorization process.

2 Likes