scottk/openafc_final
1
0
Fork 0
mirror of https://github.com/Telecominfraproject/openafc_final.git synced 2025-10-30 17:47:45 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
main
openafc_final/rcache
History

README.md

Copyright (C) 2022 Broadcom. All rights reserved.
The term "Broadcom" refers solely to the Broadcom Inc. corporate affiliate that owns the software below. This work is licensed under the OpenAFC Project License, a copy of which is included with this software program.

Rcache - AFC Response Cache

Table of contents

Operation overview

AFC Response computation takes a lot of time, hence it is beneficial to reuse previously computed AFC Response wherever possible. Rcache (Response Cache) is a mechanism for collection and retrieval of previously computed AFC Responses.

Rcache parts

Rcache consists of following parts, scattered over various corners of AFC source tree:

  • Rcache service (sources in rcache). REST API service that runs in a separate container and responsible for all write-related operations, namely:

    • Update. Writes newly computed AFC Responses to Postgres database.
    • Invalidate. Marks as invalid cache entries affected by FS (aka ULS) data change or by AFC Config changes.
    • Precompute. Invalidated cache entries recomputed in order to avoid Message Handler /RaServer delay when AFC Request will arrive next time. This is called precomputation (as it happens before AFC Request arrival).

  • Rcache client library (own and shared sources located in src/afc-packages/rcache). All AFC Services that interoperate with Rcache do it through this client library.
    Rcache library not only communicates with Rcache service, but also directly interoperates with other parts of AFC system:

    • With Postgres database - to perform cache lookup
    • With RabbitMQ - to pass computed AFC Response from Worker to Message Handler / Rat Server.

  • Rcache tool (sources located in tools/rcache). Script for Rcache testing - functional and load.
    This script is intended to be executed in its own container (for which there is Dockerfile in sources) that can be hitched to main AFC docker composition (for which there is compose_rcache.yaml in sources).
    Also it is included in Rcache service container - and may be ran from there.

Rcache counterparts

Following services interoperate with Rcache service or library:

  • Postgres database. Rcache client library performs cache lookup directly in database (on behalf of Message Handler and Rat Server), Rcache service performs write operations.

  • Message Handler / Rat Server. Uses Rcache client library to:

    • Perform Rcache lookup
    • Receive computed AFC Response sent by Worker via RabbitMQ. This route is used for synchronous non-GUI requests only. Responses on asynchronous and GUI requests passed in a legacy way - via ObjStore (without Rcache library involvement)

  • Worker. Uses Rcache client library to:

    • Pass (via RabbitMQ) computed synchronous non-GUI AFC Response to Message Handler or Rat Server
    • Write (via Rcache REST API service) computed AFC Response to Postgres database.

  • RabbitMQ. Used by Rcache client library to pass computed synchronous non-GUI AFC Responses from Worker to Message Handler or Rat Server

  • FS (aka ULS) downloader. Uses Rcache library to notify Rcache service which regions were invalidated by changes in FS data.

  • ???. For the following actions there are REST APIs in Rcache service (but no Rcache client library routines), however as of time of this writing it is not quite clear who'll drive them:

    • Invalidation on AFC Config changes.
    • Full invalidation.
    • Setting precompute quota (maximum number of simultaneous precomputations).

Associated containers and their configuration

All Rcache service, Rcache client library and, to certain extent, Rcache tool are configured by means of environment variables, passed to containers. All these variables have RCACHE_ prefix. Here are these environment variables

Name Current value Defined in Images
using it		 Meaning
RCACHE_ENABLED True .env rcache,
msghnd,
rat_server,
worker,
uls_downloader		 TRUE to use Rcache for synchronous non-GUI requests. FALSE to use legacy ObjStore-based cache
RCACHE_CLIENT_PORT 8000 .env,
docker-compose.yaml,
tools/rcache/
Dockerfile		 rcache,
rcache_tool		 Port on which Rcache REST API service is listening. Also this port is a part of RCACHE_SERVICE_URL environment variable, passed to other containers (see below)
RCACHE_
POSTGRES_
DSN		 postgresql://
postgres:postgres@
bulk_postgres/rcache		 docker-compose.yaml,
tools/rcache/
Dockerfile		 rcache,
rat_server,
msghnd,
rcache_tool		 Connection string to Postgres database that stores cache
RCACHE_
SERVICE_
URL		 http://rcache:
${RCACHE_CLIENT_PORT}		 docker-compose.yaml,
tools/rcache/
Dockerfile		 msghnd,
rat_server,
worker,
uls_downloader,
rcache_tool		 Rcache service REST API URL
RCACHE_RMQ_
DSN		 amqp://rcache:rcache@
rmq:5672/rcache		 docker-compose.yaml msghnd,
rat_server,
worker		 RabbitMQ AMQP URL Worker uses to send and msghnd/rat_server to receive AFC Response. Note that user and vhost (rcache and rcache in this URL) should be properly configured in RabbitMQ service
RCACHE_AFC_
REQ_URL		 http://msghnd:8000/
fbrat/ap-afc/
availableSpectrumInquiry?
nocache=True		 docker-compose.yaml rcache REST API Rcache service uses to launch precomputation requests. No precomputation if this environment variable empty or not defined
RCACHE_
RULESETS_
URL		 http://rat_server/fbrat/
ratapi/v1/GetRulesetIDs		 docker-compose.yaml rcache REST API URL to request list of Ruleset IDs in use. If empty or not defined default AFC distance (130km as of time of this writing) is used for spatial invalidation
RCACHE_CONFIG_
RETRIEVAL_URL		 http://rat_server/fbrat/
ratapi/v1/
GetAfcConfigByRulesetID		 docker-compose.yaml rcache REST API URL to request AFC Config by for Ruleset ID. If empty or not defined default AFC distance (130km as of time of this writing) is used for spatial invalidation
RCACHE_UPDATE_
ON_SEND		 True worker/Dockerfile worker TRUE if worker sends update request to Rcache service (this speeds up AFC request processing, but prolongs worker existence). FALSE if msghnd/rat_server send update request to Rcache service. This setting consumed by worker, but indirectly affects msghnd and rat_server services operation
RCACHE_IF_
DB_EXISTS		 leave rcache/Dockerfile rcache What to do with Rcache database if it exists when service started: leave - to leave as is, recreate - to create from scratch (losing unknown tables, like from Alembic, etc.), clean - to clean but leave other tables intact
RCACHE_
PRECOMPUTE_
QUOTA		 10 rcache/Dockerfile rcache Maximum number of parallel precompute requests. Also can be changed on the fly with Rcache service REST API

Client API

All Rcache client API presented by RcacheClient class, defined in shared rcache_client module (that should be imported) - and subsequently will (at least - should) be added there.

Data structures, used in certain interfaces of this class (as well as in Rcache REST APIs, database manipulations, etc.), are defined in rcache_models module (that also should be imported).

Among other data structures, defined in rcache_models module there is RcacheClientSettings class, containing Rcache client parameters. By default these parameters are taken from environment variables (see above), but they also can be explicitly specified.

REST API

Rcache service is FastAPI-based, so all its REST APIs are self-documented on /docs, /redoc and even /openapi.json endpoints (see https://fastapi.tiangolo.com/tutorial/metadata/).

Here is one possible way to reach this trove of knowledge (uppercase are values you need to figure out):

  1. Start AFC docker compose.
    There are no sane ways of doing it - pick your favorite insane.
  2. Inspect rcache docker container:
    docker container inspect AFCPROJECTNAME_rcache_INSTANCEINDEX
  3. Find rcache container IP in host file system:
    In printed JSON it is at [0]["NetworkSettings"]["Networks"][DEFAULTNETWORKNAME]["IPAddress"]
  4. Do ssh or plink local port forwarding from your client machine to rcache container IP, port 8000 (or whatever RCACHE_CLIENT_PORT is set to):
    SSHORPLINK -L LOCALPORT:RCACHEIPADDRESS:8000
  5. On client machine browse to localhost:LOCALPORT/redoc

rcache_tool.py - test and manipulation tool

Rcache tool requires certain uncommon Python modules to be installed (pydantic, aiohttp, sqlalchemy, tabulate). Also it uses shared Rcache sources. Hence this script supposed to be run from container.

Dockerfile for this container can be found in tools/rcache - it should be built from top level AFC checkout directory. To run properly this container should be connected to AFC internal network. This might be achieved with docker run --network NETWORKNAME ... or by hitching this container to AFC docker compose (tools/rcache/compose_rcache.yaml may be used for this).

Besides its own container this script exists in Rcache service container - and may be ran from there.

General invocation format: ./rcache_tool.py SUBCOMMAND PARAMETERS

List of available subcommands may be obtained with
./rcache_tool.py --help

Help on individual subcommand parameters may be obtained with
./rcache_tool.py help SUBCOMMAND

Some practical use cases (all examples assumed to be executed from rcache container, so rcache service location is guessed automagically):

  • Print service status. Once:
    ./rcache_tool.py status
    ... and once per second (e.g. in parallel with some test): ./rcache_tool.py status --interval 1

  • Disable cache invalidation (e.g. by ULS downloader). Useful to prevent ULS Downloader's influence on performance testing:
    ./rcache_tool.py invalidate --disable
    ... and re-enable it back:
    ./rcache_tool.py invalidate --enable
    Restoring original state is essential, as enable/disable state is persisted in database

  • Invalidate all cache AND prevent precomputer from its re-validation:
    ./rcache_tool.py precompute --disable
    ./rcache_tool.py invalidate --all
    ... and restore precomputation back:
    ./rcache_tool.py precompute --enable
    Restoring original state is essential, as enable/disable state is persisted in database

  • Disable caching:
    ./rcache_tool.py precompute --disable
    ./rcache_tool.py update --disable
    ./rcache_tool.py invalidate --all
    ... and restore status quo ante:
    ./rcache_tool.py precompute --enable
    ./rcache_tool.py update --enable
    Restoring original state is essential, as enable/disable state is persisted in database

  • Do Rcache update stress test (afc_load_tool.py also can do it), parallel writing from 20 streams:
    ./rcache_tool.py mass_fill --max_idx 1000000 --threads 20

Database schema

Rcache uses database, consisting of two tables.

Rcache table

Table name is aps, each of its rows represent data on a single AP. It has following columns:

Column Is
primary
key		 Has
index		 Data
type		 Comment
serial_number Yes Yes String AP Serial number
rulesets Yes Yes String Request Ruleset IDs, concatenated with `
cert_ids Yes Yes String Request Certification IDs, concatenated with `
state No Yes Enum
(Valid,
Invalid,
Precomp)		 Row state. Valid - may be used, Invalid - invalidated, to be precomputed, Precomp - precomputation is in progress
config_ruleset No Yes String Ruleset ID used for computation. Used for invalidation by Ruleset ID (i.e. by AFC Config change)
lat_deg No Yes Float AP Latitude as north-positive degree. Used for spatial invalidation (on FS data change)
lon_deg No Yes Float AP Longitude as east-positive degree. Ditto.
last_update No Yes DateTime Timetag of last update - eventually may be used for removal of old rows
req_cfg_digest No Yes String Hash, computed over AFC Request and AFC Config. Used as key for cache lookup
validity_period_sec No No Float Validity period of original response. Used to compute expiration time of response retrieved from cache
request No No String AFC Request as string. Used for precomputation
response No No String AFC Response as string. The meat of all this story

Switches table

Table name is switches, each of its rows represent enable/disable setting. It has following columns:

Column Comment
name Setting name. This is a primary key. As of time of this writing, names were: Update, Invalidate, Precompute
state True if enabled, False if disabled. Default is True