This is Beta Software. Finished releases are published directly to svn:dist/release.

7.1. Overview of the code

Up: 7. Developer guide

Prev: 7. Developer guide

Next: 7.2. How to contribute

Sections:

Introduction

This page is a high level view of the ATR code. References to symbols in this section are given without their atr. prefix, for brevity, and are linked to the respective source code. You should understand e.g. server.app (app) to mean atr.server.app.

Hypercorn and ASGI

ATR is an ASFQuart application, running in Hypercorn. The entry point for Hypercorn is server.app (app), which is then called with a few standard arguments as per the ASGI specification. We create the app object using the following code:

app = create_app(config.get())

The server.create_app (create_app) function performs a lot of setup, and if you're interested in how the server works then you can read it and the functions it calls to understand the process further. In general, however, when developing ATR we do not make modifications at the ASFQuart, Quart, and Hypercorn levels very often.

Routes and database

Users request ATR pages over HTTPS, and the ATR server processes those requests in route handlers. Most of those handlers are in get and post, with some helper code in shared. What each handler does varies, of course, from handler to handler, but most perform at least one access to the ATR SQLite database.

The path of the SQLite database is configured in config.AppConfig.SQLITE_DB_PATH (SQLITE_DB_PATH) by default, and will usually appear as state/atr.db with related shm and wal files. We do not expect ATR to have so many users that we need to scale beyond SQLite.

We use SQLModel, an ORM utilising Pydantic and SQLAlchemy, to create Python models for the ATR database. The core models file is models.sql. The most important individual SQLite models in this module are Committee (Committee), Project (Project), and Release (Release).

It is technically possible to interact with SQLite directly, but we do not do that in the ATR source. We use various interfaces in db for reads, and interfaces in storage for writes. We plan to move the db code into storage too eventually, because storage is designed to have read components and write components. There is also a legacy db.interaction module which we plan to migrate into storage.

These interfaces, including route handlers in get, post, and shared, along with models.sql and storage, are where the majority of activity happens when developing ATR.

User interface

ATR provides a web interface for users to interact with the platform, and the implementation of that interface is split across several modules. The web interface uses server-side rendering almost entirely, where HTML is generated on the server and sent to the browser.

The template system in ATR is Jinja2, always accessed through the ATR template module. Template files in Jinja2 syntax are stored in templates/, and route handlers render them using the asynchronous template.render (render) function.

Template rendering can be slow if templates are loaded from disk on every request. To address this, we use preload to load all templates into memory before the server starts serving requests. The preload.setup_template_preloading (setup_template_preloading) function registers a startup hook that finds and caches every template file.

The ATR user interface includes many HTML forms. We use Pydantic for form handling, accessed through the ATR form module. The form.Form (Form) base class extends pydantic.BaseModel to provide form-specific functionality. Each form field is defined using Pydantic type annotations along with the form.label (label) function for metadata like labels and documentation. Validation happens automatically through Pydantic's validation system, and custom validators can be added using Pydantic's @model_validator decorator. Forms are rendered in templates, but the ATR form module also provides programmatic rendering through the form.render (render) function that generates HTML styled with Bootstrap.

In addition to templates, we sometimes need to generate HTML programmatically in Python. For this we use htpy, another third party library, for building HTML using Python syntax. The ATR htm module extends htpy with a Block (Block) class that makes it easier to build complex HTML structures incrementally. Using htpy means that we get type checking for our HTML generation, and can compose HTML elements just like any other Python objects. The generated HTML can be embedded in Jinja2 templates or returned directly from route handlers.

Refer to the full user interface development guide for more information about this topic.

Scheduling and tasks

Many operations in ATR are too slow to run during an HTTP request, so we run them asynchronously in background worker processes. The task scheduling system in ATR is built from three components: a task queue stored in the SQLite database, a worker manager that spawns and monitors worker processes, and the worker processes themselves that claim and execute tasks.

The task queue is stored in a table defined by the Task (Task) model in models.sql. Each task has a status, a type, arguments encoded as JSON, and metadata such as when it was added and which user created it. When route handlers need to perform slow operations, they create a new Task row with status QUEUED and commit it to the database.

The ATR manager module provides the WorkerManager (WorkerManager) class, which maintains a pool of worker processes. When the ATR server starts, the manager spawns a configurable number of worker processes and monitors them continuously. The manager checks every few seconds whether workers are still running, whether any tasks have exceeded their time limits, and whether the worker pool needs to be replenished. If a worker process exits after completing its tasks, the manager spawns a new one automatically. If a task runs for too long, the manager terminates it and marks the task as failed. Worker processes are represented by WorkerProcess (WorkerProcess) objects.

The ATR worker module implements the workers. Each worker process runs in a loop. It claims the oldest queued task from the database, executes it, records the result, and then claims the next task atomically using an UPDATE ... WHERE statement. After a worker has processed a fixed number of tasks, it exits voluntarily to help to avoid memory leaks. The manager then spawns a fresh worker to replace it. Task execution happens in the _task_process (_task_process) function, which resolves the task type to a handler function and calls it with the appropriate arguments.

Tasks themselves are defined in the ATR tasks directory. The tasks module contains functions for queueing tasks and resolving task types to their handler functions. Task types include operations such as importing keys, generating SBOMs, sending messages, and importing files from SVN. The most common category of task is automated checks on release artifacts. These checks are implemented in tasks/checks/, and include verifying file hashes, checking digital signatures, validating licenses, running Apache RAT, and checking archive integrity.

API

The ATR API provides programmatic access to most ATR functionality. API endpoints are defined in api, and their URL paths are prefixed with /api/. The API uses OpenAPI for documentation, which is automatically generated from the endpoint definitions and served at /api/docs. Users send requests with a JWT created from a PAT. The jwtoken module handles issuing and verifying these tokens. API endpoints that require authentication declare a header auth scheme on their @api.typed route, which verifies the JWT via jwtoken.authenticate.

API request and response models are defined in models.api using Pydantic. Each endpoint has an associated request model that validates incoming data, and a response model that validates outgoing data. The API returns JSON in all cases, with appropriate HTTP status codes.

Other important interfaces

ATR uses ASF OAuth for user login, and then determines what actions each user can perform based on their committee memberships. The ATR principal module handles authorization by checking whether users are members of relevant committees. It queries and caches LDAP to get committee membership information. The Authorisation (Authorisation) class provides methods to check whether a user is a member of a committee or a project participant, which can result in different levels of access.

The server configuration in config determines a lot of global state, and the util module contains lots of useful code which is used throughout ATR.