Export all scan results (from a single report or multiple) when then are more than 1000 results

How To
1

GVM versions

gsad: 21.4.4
gvmd: 21.4.5
openvas-scanner: 21.4.4
gvm-libs: 21.4.4

Environment

Operating system: Linux
Kernel: 5.16.0-amd64
Installation method / source: debian package

The problem

I’d like to know how to get ALL the results from the already-complete scan with more than 1000 results in xml format.
At this point any way will do - using GSA, or the python-gvm.

I understand that 1000 is a hard-limit for any entities, but getting scan results is a pretty common task,
so WTF - why is there no sane documentation on how to accomplish this?! Moreover - GSA doesn’t notify in any way that not all the results have been included. This is outrageous.

Things tried so far

  • ignore_pagination=1 levels=hmlg min_qod=0 filter
  • using first=1000 to get the rest of the results in filter
  • rows=-1 filter
2

@Yurii does --max-rows 0 fix the issue? or --max-rows 3000 or similar?

sounds like hard coded value and cannot find it from sourcecodes

Eero

3

@Eero
Could you please specify the script/tool that is supposed to take the key (–max-rows)?

4

@Yurii sounds like value is hardcoded into sourcecode. so, it means that you can only get 1000 results for one host. that is “enought for most of people”…

Eero

5

@Eero For one host - sure. My problem is that I cannot get more than 1000 results from the scan, which was performed on more than 300 hosts. Whatever I’ve tried the export yields 1000 results, which is about 60% of all results, available in the report.

6

@Yurii that is feature, not a bug :slight_smile:

Eero

7

@Yurii maybe it’s possible to modify source to skip this hardcoded max value. this means recompiling some parts of software.

Eero

8

Hi,

to get more then 1000 results you need to request a report with setting ignore_pagination to 1. For example : <get_reports report_id="..." ignore_pagination="1" details="1" filter="levels=hmlg min_qod=0" />. It’s not possible to request more then 1000 items with <get_results/>.

2 Likes
9

@bricks This sounds like great news. Could you please be more specific:

  • is the example you’ve provided meant to be used as a reference for raw api request?
  • can the same be done in gvm-python - I’ve tried get_reports(), but it doesn’t return the expected ‘report/results’ tag
  • can the same be done in gsa gui?
10

To allow more then 1000 items, you can run

gvmd --modify-setting 76374a7a-0569-11e6-b6da-28d24461215b --value 10000 as your user that is run gvmd like gvm

ref: https://github.com/greenbone/gvmd/blob/7862bd54a0c7524524e6460dad70f155dd30510d/src/manage_sql.h#L113

but this will affect also some other things and makes it slower.

11

@bricks example was for the gvm-cli raw xml request.

get_report() should return what you try with get_reports()?

yes, you can export the xml.

1 Like
12

Yes

Yes of course. See the get_report API docs.

gmp.get_report(report_id, filter_string="...", ignore_pagination=True, details=True)

The UI is doing exactly this. You can take a look at the network requests in your browser’s console.

1 Like
13

@Dexus @bricks
Thank you. Got it - gvm-cli xml req works nicely.

1 Like
15

Please share your command that it’s worked. @Yurii

16

Hi @Paiii3 did you try the commands mentioned by @bricks in Export all scan results (from a single report or multiple) when then are more than 1000 results - #8 by bricks - Greenbone Community Edition - Greenbone Community Portal?

If you want to use python-gvm, you will need to set the same variables to the same values. See API.

E.g.:

result = gmp.get_reports("1234-5678-90ab-cdef1234", details=True, ignore_pagination=True)
17

Thanks for your help.

18

I tried to scrape my Greenbone Community site using developer tools. Not sure if this might work.

https://<yourGreenbone?>/gmp?token=???&cmd=get_results&details=1&filter=sort-reverse%3Dseverity%20rows%3D100000000000000000000000

image

The output is XML so you could pipe into something else for parsing?

19

If it’s still relevant:

get_report_cmd="<get_reports report_id=\"$report_id\" ignore_pagination=\"1\" details=\"1\" filter=\"$filter\" />"
gvm-cli \
    --gmp-username "$user" \
    --gmp-password "$pass" \
    socket \
    --xml "$get_report_cmd"

mind you will need to include yout preferred authentication, user/pass is just as an example

2 Likes
20

Hello, how to export to csv with 15000resulsts?Or someone created a converter from xml to csv?
I can upload xml like this
gvm-cli --gmp-username user --gmp-password password --timeout 900 socket --socketpath /var/run/gvmd/gvmd.sock --xml “<get_reports report_id="ID" details="1" ignore_pagination="1" format_id="a994b278-1f62-11e1-96ac-406186ea4fc5"/>” >test.xml
The data is all there, but the rows are divided into sections according to an xml scheme unknown to me and I cannot match the necessary columns, as in csv. I only need
IP,Hostname,CVSS,Severity,NVT Name, Summary, Specific Result, NVT OID, CVEs, Task Name, Timestamp, Impact, Solution, how to do this?
sorry for the my english, i use google translate…

21

Hi. Gsa (web-interface) has the option to export directly to csv. Alternatively you can export a small set, take a look at the xml scheme and convert to scv using python - this would be a really small and straightforward script.