Important: IP and bandwith limitations on Community Feed

Hi all!

Unfortunately we’ve had to reimplement some limits on usage of the Greenbone Community Feed for technical reasons.

  1. Some people are misusing and hitting the feed server pretty hard with much too frequent resyncing, in turn overloading it (and then almost no one can use it).

  2. Way too many connection attempts are being made to our systems, which leads to unresponsive systems and heavy loads.

To keep the service available for every user, limits are set to reducing connection per IP and bandwidth.

It’s not the first time this has happened (as is the nature of service providers) but extra sad when open source projects are affected. Providing the community servers generates reasonable maintenance time and costs, currently covered completely by our sponsor Greenbone.

Important info: the feed is only updated once a day, so if it’s already updated, there should be no need to sync it again. We’re also looking into alternatives for more robust feed distribution. We apologize for the temporary limitation and difficulty in reaching the feed and we thank you for understanding and support.

3 Likes

DeeAnn,

In trying to deploy a new scanner with a fresh installation of GSE (spun up a clean VM and install of GSE yesterday), I’m not able to start the GVM service as SCAP data cannot be acquired from the feed server, or at least that’s how I’m interpreting the error and failures I’m encountering. At first, it was NVTs but eventually, I succeeded in a feed sync long enough to acquire those. As this is a fresh installation, and assuming I’m interpreting the issue correctly, I’m dead in the water until I can run a feed sync that doesn’t fail to acquire SCAP data. I’ve been periodically running

sudo runuser -u _gvm – greenbone-feed-sync --type SCAP

but it times out every time. How do you recommend I proceed, taking into consideration that I can’t even start GVM for the first time without a successful feed sync?

P.S.- I’m still a bit shaky on the difference between GSE, GVM, GSF, GCF, and the rest of your products and their naming conventions, some of which I understand are open source and not maintained or kept up by you guys. So forgive me for confusing the names and products and know I do understand you can’t speak for some of the software and tools mentioned above.

Same here. I been trying for days to get the feeds downloaded once, so I can start GVM. This is very frustrating.

I have the same problem.

Hi everyone,

Yep, it’s still having issues and technical operations is looking into it. We’ll let you know as we find out. Thanks!

2 Likes

As an update, we had to block some traffic that was showing extreme over usage (or possible attacks) of the feed. Hopefully that will help for now, but we are still looking into ways to continuously improve it.

3 Likes

Now the server looks pretty stable ,you might wanna sync again. Please ensure one connection per src-IP, if you are behind a NAT or shared connection, get a different internet source.

We firewalled the abusive users with more then 10k DoS attack.

4 Likes

Thank you for the update and hard work. I was able to update my deployed scanners.

For the sanity of the service, excessive users how are trying to do more then 1000 connection within 24h will from now on be blocked for the community feed. That is for the greater good of all honest users.

2 Likes

We still face DoS to our services, i activated SYN Cookies,
Your Setup needs to support SYN Cookies otherwise you can not sync any more.

1 Like

DeeAnn,

I am currently trying to do a project for a class and I am unable to complete the command sudo runuser -u _gvm – greenbone-feed-sync --type CERT. Sometimes it looks like it starts to run and then just winds up timing out. I have destroyed my Kali Box twice in order to get this to run, but my professor just says to keep retrying the command over the days. The deadline for my project is coming up and I can’t seem to get it up to update. It keeps giving me the rsync: [Receiver] failed to connect to feed.community.greenbone.net (45.135.106.142): Connection timed out (110) error over and over no matter if I try it back to back, wait a couple of hours, or wait a day in between.

Can you please provide some guidance so I can pass this stage and continue to see if my gvm-check-setup was configured correctly?

Thank you so much in advance.

yes, this is very frustating. I just installed a new kali box with greenbone, but cannot start it because I cannot update the feeds.
@Lukas : I think my box supports syn_cookies; however, how can we be sure?

As a suggestion: maybe it’s possible to implement (some kind of) a free key / subscription ?
Hope you can solve these problems as I really enjoy using Greenbone Community Edition…

Hi @asawula ,

Maybe you should check just now, as it seems to be working now…!

If some one runs a installation with 10.000 Units trying to sync simultaneously he will definitively overload our service. Bedsides this we see attacks to our service so the total sync bandwidth gets down on the back of everyone. We implement some counter measure and then again we got a new attack. Now we wen´t from SYN Cookies to random early drop and total limit of 150 simultaneous connections.

As you can see the traffic gets down due to attacks, and yes you are absolutely right we need to think about a free key that can sync once a day with a limited amount of installation to act fair to the community. The plan was to keep the entry level low and enable a anonymous service but it seems that some take unfair advantage from this.


     2021-10-18    64.58 GiB |    2.07 TiB |    2.13 TiB |  216.94 Mbit/s
     2021-10-19    69.21 GiB |    1.68 TiB |    1.75 TiB |  177.94 Mbit/s
     2021-10-20    71.70 GiB |    1.71 TiB |    1.78 TiB |  181.56 Mbit/s
     2021-10-21    91.24 GiB |    1.81 TiB |    1.90 TiB |  193.18 Mbit/s
     2021-10-22    64.23 GiB |    1.78 TiB |    1.84 TiB |  187.54 Mbit/s
     2021-10-23    57.36 GiB |    1.47 TiB |    1.52 TiB |  155.13 Mbit/s
     2021-10-24    58.24 GiB |    1.83 TiB |    1.89 TiB |  192.49 Mbit/s
     2021-10-25    62.04 GiB |    1.98 TiB |    2.04 TiB |  207.68 Mbit/s
     2021-10-26    59.62 GiB |    1.78 TiB |    1.84 TiB |  187.46 Mbit/s
     2021-10-27    55.03 GiB |    1.50 TiB |    1.55 TiB |  157.95 Mbit/s
     2021-10-28    71.29 GiB |    1.41 TiB |    1.48 TiB |  150.65 Mbit/s
     2021-10-29    54.74 GiB |    1.34 TiB |    1.40 TiB |  142.23 Mbit/s
     2021-10-30    48.73 GiB |    1.29 TiB |    1.34 TiB |  136.43 Mbit/s
     2021-10-31    51.59 GiB |    1.61 TiB |    1.66 TiB |  169.25 Mbit/s
     2021-11-01    43.60 GiB |    1.19 TiB |    1.23 TiB |  125.35 Mbit/s
     2021-11-02    37.57 GiB |  624.34 GiB |  661.91 GiB |   65.81 Mbit/s
     2021-11-03    27.65 GiB |  946.15 GiB |  973.79 GiB |   96.82 Mbit/s
     2021-11-04    14.49 GiB |  601.45 GiB |  615.94 GiB |   61.24 Mbit/s
     2021-11-05    38.85 GiB |    0.98 TiB |    1.02 TiB |  103.37 Mbit/s
     2021-11-06     3.45 GiB |  225.12 GiB |  228.57 GiB |   22.72 Mbit/s
     2021-11-07     2.04 GiB |   93.49 GiB |   95.53 GiB |   20.26 Mbit/s

Nope I still get the rsync error.

Can we / someone help with load balancing or similar ?

What about using a CDN (CloudFlare, CloudFront, Akamai, etc…)? The feeds are static content, changing once per day or less.

We are working actively on a solution, first upgrade our firewall to get us better protection and then a load-balancer to more then two servers. But if someone abuse our service even a CDN will not help and comes with a heavy price tag.

Hi all,

gvm-check-setup 21.4.3
Test completeness and readiness of GVM-21.4.3
Step 1: Checking OpenVAS (Scanner)…
OK: OpenVAS Scanner is present in version 21.4.3.
OK: Server CA Certificate is present as /var/lib/gvm/CA/servercert.pem.
Checking permissions of /var/lib/openvas/gnupg/*
OK: _gvm owns all files in /var/lib/openvas/gnupg
OK: redis-server is present.
OK: scanner (db_address setting) is configured properly using the redis-server socket: /var/run/redis-openvas/redis-server.sock
OK: redis-server is running and listening on socket: /var/run/redis-openvas/redis-server.sock.
OK: redis-server configuration is OK and redis-server is running.
OK: _gvm owns all files in /var/lib/openvas/plugins
OK: NVT collection in /var/lib/openvas/plugins contains 77656 NVTs.
Checking that the obsolete redis database has been removed
OK: No old Redis DB
OK: ospd-OpenVAS is present in version 21.4.3.
Step 2: Checking GVMD Manager …
OK: GVM Manager (gvmd) is present in version 21.4.4.
Step 3: Checking Certificates …
OK: GVM client certificate is valid and present as /var/lib/gvm/CA/clientcert.pem.
OK: Your GVM certificate infrastructure passed validation.
Step 4: Checking data …
ERROR: SCAP DATA are missing.
FIX: Run the SCAP synchronization script greenbone-feed-sync.
sudo runuser -u _gvm – greenbone-feed-sync --type SCAP.

ERROR: Your GVM-21.4.3 installation is not yet complete!

Please follow the instructions marked with FIX above and run this
script again.