Installation#

This section covers the installation steps of the sofware.

System dependencies#

You need Poetry installed, see the install guide.

Prerequisites#

You need to have Redis and Kvrocks built from source available on your machine. The standard scripts of vulnerability-lookup expect you to have a final directory structure like so:

/vulnerability-lookup (this codebase)
/redis
/kvrocks

Specifically, ./vulnerability-lookup/cache/run_redis.sh runs ./redis/src/redis-server and ./vulnerability-lookup/storage/run_kvrocks.sh runs ./kvrocks/build/kvrocks (or starts the apache/kvrocks docker). The vulnerability-lookup launcher provides configuration for both Redis and Kvrocks.

Redis#

Redis: An open source (BSD licensed), in-memory data structure store, used as a database, cache and message broker.

You need to have redis cloned and installed in the same directory you clone this template in: this repository and redis must be in the same directory, and not redis cloned in this directory. See this guide.

In order to compile and test redis, you will need a few packages:

sudo apt-get update
sudo apt install build-essential tcl

git clone https://github.com/redis/redis.git
cd redis
git checkout 7.2
make
# Optionally, you can run the tests:
make test
cd ..

Kvrocks#

Kvrocks is a distributed key value NoSQL database that uses RocksDB as storage engine and is compatible with Redis protocol. Kvrocks intends to decrease the cost of memory and increase the capability while compared to Redis.

Note

Kvrocks should be installed from the source, and the repository must be in one directory up as the one you will be cloning vulnerability-lookup into. It is also possible to make the apache/kvrocks docker container available to use.

In order to compile kvrocks, you will need a few packages:

sudo apt-get update
sudo apt install git gcc g++ make cmake autoconf automake libtool python3 libssl-dev

git clone --recursive https://github.com/apache/kvrocks.git
cd kvrocks
git checkout v2.9.0
./x.py build
cd ..

Import CSAF sources#

Build the support tools.
(Note this requires golang >= 1.23)
Make sure the downloader exists:

$ (git::main) ./bin-linux-amd64/csaf_downloader -h
Usage:
csaf_downloader [OPTIONS] domain...

Application Options:
-d, --directory=DIR                             DIRectory to store the downloaded files in
    --insecure                                  Do not check TLS certificates from provider
    --ignore_sigcheck                           Ignore signature check results, just warn on mismatch
    --client_cert=CERT-FILE                     TLS client certificate file (PEM encoded data)
    --client_key=KEY-FILE                       TLS client private key file (PEM encoded data)
    --client_passphrase=PASSPHRASE              Optional passphrase for the client cert (limited, experimental, see doc)
    --version                                   Display version of the binary
-n, --no_store                                  Do not store files
-r, --rate=                                     The average upper limit of https operations per second (defaults to unlimited)
-w, --worker=NUM                                NUMber of concurrent downloads (default: 2)
-t, --time_range=RANGE                          RANGE of time from which advisories to download
-f, --folder=FOLDER                             Download into a given subFOLDER
-i, --ignore_pattern=PATTERN                    Do not download files if their URLs match any of the given PATTERNs
-H, --header=                                   One or more extra HTTP header fields
    --validator=URL                             URL to validate documents remotely
    --validator_cache=FILE                      FILE to cache remote validations
    --validator_preset=PRESETS                  One or more PRESETS to validate remotely (default: [mandatory])
-m, --validation_mode=MODE[strict|unsafe]       MODE how strict the validation is (default: strict)
    --forward_url=URL                           URL of HTTP endpoint to forward downloads to
    --forward_header=                           One or more extra HTTP header fields used by forwarding
    --forward_queue=LENGTH                      Maximal queue LENGTH before forwarder (default: 5)
    --forward_insecure                          Do not check TLS certificates from forward endpoint
    --log_file=FILE                             FILE to log downloading to (default: downloader.log)
    --log_level=LEVEL[debug|info|warn|error]    LEVEL of logging details (default: info)
-c, --config=TOML-FILE                          Path to config TOML file

Help Options:
-h, --help                                      Show this help message

Add the full path to the downloader in config/generic.json key csaf_downloader_path

Clone the source code#

Clone Vulnerability-Lookup with the submodules.

git clone --recursive https://github.com/vulnerability-lookup/vulnerability-lookup.git

Installation#

From the directory you just cloned, run:

poetry install

Initialize the .env file:

echo VULNERABILITYLOOKUP_HOME="`pwd`" >> .env

Note

VULNERABILITYLOOKUP_HOME is the name you set in vulnerability-lookup/default/__init__.py

Initialize the submodules (as the repositories are quite large, it can take sometime):

git submodule update --init

Configuration#

Generic configuration#

Copy and adapt the following config files:

cp config/generic.json.sample config/generic.json
cp config/logging.json.sample config/logging.json
cp config/website.py.sample config/website.py
cp config/stream.json.sample config/stream.json

And configure them according to your needs.

Modules#

Request API keys for NVD and VARIoT here:

Copy the module file and update the file with these API keys:

cp config/modules.cfg.sample config/modules.cfg

User accounts#

Initialize the database used for the management of user accounts, comments and bundles. user_accounts must be set to true in the config/generic.json configuration file.

Create a PostgreSQL user and a database:

$ sudo apt install postgresql
$ sudo -u postgres createuser -P <username>
Enter password for new role: <password>
$ sudo -u postgres createdb <database> -O <username>

Then configure config/website.py with <username> and <password> in the DB_CONFIG_DICT, and set DATABASE_NAME to the right database (default vulnlookup).

Initialize the database:

$ poetry run flask --app website.app db_init # initializes the databse
$ poetry run flask --app website.app db stamp head # add alembic_version table to the database and point to the most recent version

After creating the database, you can proceed to create the first admin user:

$ poetry run flask --app website.app create_admin --login admin --email admin@example.org --password <password>

Optionally, you can import the OSI licenses and a list of programming languages:

$ poetry run flask --app website.app import_osi_approved_licenses
$ poetry run flask --app website.app import_languages

Initialize the local GCVE registry:

poetry run flask --app website.app update_gcve_registry

Usage#

Start the tool (as usual, from the directory):

poetry run start

You can stop it with:

poetry run stop

With the default configuration, you can access the web interface on http://0.0.0.0:10001, where you will find the API and can start playing with it.

Update the tool:

poetry run update

Launching the website with systemd#

Note

This is an alternative method. The website will be automatically launched with the start command.

Create a file /etc/systemd/system/vulnerability-lookup-web.service with the following contents:

[Unit]
Description=Vulnerability-Lookup webservice
After=network.target

[Service]
User=<system user used to install Vulnerability-Lookup>
Group=<group of the user used to install Vulnerability-Lookup>
WorkingDirectory=<path to the directory where you cloned the repository>
Environment=PATH="<path-of-the-virtualenv>/bin/:/usr/bin"
ExecStart=/bin/bash -c "run_backend --start ; start_website"
ExecStop=/bin/bash -c "run_backend --stop"

[Install]
WantedBy=multi-user.target

$ sudo systemctl daemon-reload
$ sudo systemctl enable vulnerability-lookup-web.service
$ sudo systemctl start vulnerability-lookup-web.service
$ systemctl status vulnerability-lookup-web.service

Installation

Contents

Installation#

System dependencies#

Prerequisites#

Redis#

Kvrocks#

Import CSAF sources#

Clone the source code#

Installation#

Configuration#

Generic configuration#

Modules#

User accounts#

Usage#

Launching the website with systemd#