Docs updated: python scripts added to generate stencil and example code listings; doc-build to ci added

This commit is contained in:
Nicolas Kruse 2025-12-26 23:49:22 +01:00
parent 069f79094c
commit c5d3fba531
13 changed files with 318 additions and 83 deletions

View File

@ -308,3 +308,28 @@ jobs:
echo "Updating existing release for $TAG"
gh release upload "$TAG" release/* --clobber
fi
build-docs:
#if: github.event_name == 'push'
needs: [build_stencils, build-ubuntu, build-windows, build-arm64, build-armv6, build-armv7]
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Download artifacts
uses: actions/download-artifact@v4
with:
path: build/tmp
- name: Set up Python
uses: actions/setup-python@v3
with:
python-version: "3.x"
- name: Install package and dependencies
run: pip install sphinx pydata_sphinx_theme sphinx-autodoc-typehints myst-parser
- name: Build Docs
run: |
make html
touch ./build/html/.nojekyll

View File

@ -14,6 +14,17 @@ help:
.PHONY: help Makefile
prepare-docs:
python $(SOURCEDIR)/generate_class_list.py --api-dir $(SOURCEDIR)/api
python $(SOURCEDIR)/extract_section.py --readme $(SOURCEDIR)/../../README.md --build-dir $(BUILDDIR)
python $(SOURCEDIR)/stencil_doc.py --input $(SOURCEDIR)/../../build/stencils.c --asm-pattern "$(SOURCEDIR)/../../build/tmp/runner-linux-*/stencils.asm" --output $(BUILDDIR)/stencils.md
python $(SOURCEDIR)/example_asm.py --input $(SOURCEDIR)/../../tools/make_example.py --asm-pattern "$(SOURCEDIR)/../../build/tmp/runner-linux-*/example.asm" --output $(BUILDDIR)/compiled_example.md
# Build documentation (generate API and extract sections first)
html: prepare-docs
@$(SPHINXBUILD) -M html "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
# Catch-all target: route all unknown targets to Sphinx using the new
# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).
%: Makefile

View File

@ -25,6 +25,13 @@ if errorlevel 9009 (
if "%1" == "" goto help
md %BUILDDIR%
python %SOURCEDIR%\generate_class_list.py --api-dir %SOURCEDIR%\api
python %SOURCEDIR%\extract_section.py --readme %SOURCEDIR%/../../README.md --build-dir %BUILDDIR%
python %SOURCEDIR%\stencil_doc.py --input "%SOURCEDIR%/../../build/stencils.c" --asm-pattern "%SOURCEDIR%/../../build/tmp/runner-linux-*/stencils.asm" --output %BUILDDIR%/stencils.md
python %SOURCEDIR%\example_asm.py --input "%SOURCEDIR%/../../tools/make_example.py" --asm-pattern "%SOURCEDIR%/../../build/tmp/runner-linux-*/example.asm" --output %BUILDDIR%/compiled_example.md
%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
goto end

7
docs/source/LICENSE.md Normal file
View File

@ -0,0 +1,7 @@
---
orphan: true
---
# License
```{include} ../../LICENSE
```

View File

@ -1,43 +1,12 @@
# How it works
The compilation step starts with tracing the Python code to generate an acyclic directed graph (DAG) of variables and operations. The code can contain functions, closures, branching, and so on, but conditional branching is only allowed when the condition is known at tracing time (a `cp.iif` function exists to work around this). In the next step, this DAG is optimized and linearized into a sequence of operations. Each operation is mapped to a precompiled stencil or a combination of several stencils. A stencil is a piece of machine code with placeholders for memory addresses pointing to other code or data. The compiler generates patch instructions that fill these placeholders with the correct memory addresses.
After compilation, the binary code built from the stencils, the constant data, and the patch instructions is handed to the runner for execution. The runner allocates memory for code and data, copies both into place, applies the patch instructions, and finally executes the code.
The C code for a very simple stencil can look like this:
```c
add_float_float(float arg1, float arg2) {
result_float_float(arg1 + arg2, arg2);
}
# Compiler
```{toctree}
:maxdepth: 1
:hidden:
stencil_doc
example_asm
```
The call to the dummy function `result_float_float` ensures that the compiler keeps the result and the second operand in registers for later use. The dummy function acts as a placeholder for the next stencil. Copapy uses two virtual registers, which map on most relevant architectures to actual hardware registers. Data that cannot be kept in a register is stored in statically allocated heap memory. Stack memory may be used inside some stencils, but its usage is essentially fixed and independent of the Copapy program, so total memory requirements are known at compile time.
The machine code for the function above, compiled for x86_64, looks like this:
```nasm
0000000000000000 <add_float_float>:
0: f3 0f 58 c1 addss %xmm1,%xmm0
4: e9 00 00 00 00 jmp 9 <.LC1+0x1>
5: R_X86_64_PLT32 result_float_float-0x4
```{include} ../build/compiler.md
```
Based on the relocation entry for the `jmp` to the symbol `result_float_float`, the `jmp` instruction is stripped when it is the last instruction in a stencil. Thus, a Copapy addition operation results in a single instruction. For stencils containing multiple branch exits, only the final `jmp` is removed; the others are patched to jump to the next stencil.
For more complex operations - where inlining is less useful - stencils call a non-stencil function, such as in this example:
```nasm
0000000000000000 <sin_float>:
0: 48 83 ec 08 sub $0x8,%rsp
4: e8 00 00 00 00 call 9 <sin_float+0x9>
5: R_X86_64_PLT32 sinf-0x4
9: 48 83 c4 08 add $0x8,%rsp
d: e9 00 00 00 00 jmp 12 <.LC0+0x2>
e: R_X86_64_PLT32 result_float-0x4
```
Unlike stencils, non-stencil functions are not stripped and do not need to be tail-call-optimizable.
Non-stencil functions and constants are stored together with the stencils in an ELF object file for each supported CPU architecture. The required non-stencil functions and constants are bundled during compilation. The compiler includes only the data and code required for the specific program.
The whole compilation process is independent of the actual instruction set. It relies purely on relocation entries and symbol metadata from the ELF file generated by the C compiler.
A full listing of all stencils with machine code for all architectures from latest build is here available: [Stencil overview](stencil_doc.md). The compiler output for a full example program from latest compiler build is here available: [Example program](example_asm).

View File

@ -0,0 +1,3 @@
# Example program
```{include} ../build/compiled_example.md
```

View File

@ -0,0 +1,67 @@
from pathlib import Path
import glob
import argparse
def build_asm_code_dict(asm_glob_pattern: str) -> dict[str, str]:
"""
Build a dictionary of assembly code for all available architectures.
Args:
asm_glob_pattern: Glob pattern to find stencils.asm files
Returns:
Dictionary mapping architecture names to their asm_code dictionaries
"""
asm_code: dict[str, str] = {}
# Find all stencils.asm files matching the pattern
asm_files = glob.glob(asm_glob_pattern)
for asm_file in asm_files:
arch_name = Path(asm_file).parent.name.replace('runner-linux-', '')
try:
with open(asm_file) as f:
asm_code[arch_name] = f.read()
print(f"Loaded assembly for {arch_name}")
except FileNotFoundError:
print(f"Warning: Assembly file not found for {arch_name}: {asm_file}")
return asm_code
# Example usage:
if __name__ == "__main__":
parser = argparse.ArgumentParser(description="Generate stencils documentation from C and assembly code")
parser.add_argument('--input', default='tools/make_example.py', help='Path to input C file')
parser.add_argument('--asm-pattern', default='build/tmp/runner-linux-*/example.asm', help='Glob pattern for assembly files')
parser.add_argument('--output', default='docs/build/compiled_example.md', help='Output markdown file path')
args = parser.parse_args()
# Build assembly code dictionary for all architectures
asm_code = build_asm_code_dict(args.asm_pattern)
with open(args.input) as f:
python_code = f.read()
md_code: str = f"""
Example program:
```python
{python_code}
```
"""
for arch in sorted(asm_code.keys()):
md_code += f"""
## {arch}
```nasm
{asm_code[arch]}
```
"""
with open(args.output, 'wt') as f:
f.write(md_code)
print(f"Generated {args.output} for {len(asm_code)} architectures")

View File

@ -1,4 +1,6 @@
import re
import argparse
import os
def extract_sections(md_text: str) -> dict[str, str]:
"""
@ -23,12 +25,19 @@ def extract_sections(md_text: str) -> dict[str, str]:
return sections
if __name__ == '__main__':
with open('README.md', 'rt') as f:
parser = argparse.ArgumentParser(description='Extract sections from README.md and generate documentation files')
parser.add_argument('--readme', type=str, default='README.md', help='README.md path')
parser.add_argument('--build-dir', type=str, default='docs/source', help='Build directory for output files (default: docs/source)')
args = parser.parse_args()
readme_path = args.readme
build_dir = args.build_dir
with open(readme_path, 'rt') as f:
readme = extract_sections(f.read())
with open('docs/source/start.md', 'wt') as f:
with open(os.path.join(build_dir, 'start.md'), 'wt') as f:
f.write('\n'.join(f"# {s}\n" + readme[s] for s in ['Copapy', 'Current state', 'Install', 'License']))
with open('docs/source/compiler.md', 'wt') as f:
f.write('# How it works\n')
with open(os.path.join(build_dir, 'compiler.md'), 'wt') as f:
f.write('\n'.join(readme[s] for s in ['How it works']))

View File

@ -5,13 +5,14 @@ import inspect
import fnmatch
from io import TextIOWrapper
import os
import argparse
def write_manual(f: TextIOWrapper, doc_files: list[str], title: str) -> None:
write_dochtree(f, title, doc_files)
def write_classes(f: TextIOWrapper, patterns: list[str], module_name: str, title: str, description: str = '', exclude: list[str] = []) -> None:
def write_classes(f: TextIOWrapper, patterns: list[str], module_name: str, title: str, description: str = '', exclude: list[str] = [], api_dir: str = 'api') -> None:
"""Write the classes to the file."""
module = importlib.import_module(module_name)
@ -27,7 +28,7 @@ def write_classes(f: TextIOWrapper, patterns: list[str], module_name: str, title
write_dochtree(f, title, classes)
for cls in classes:
with open(f'docs/source/api/{cls}.md', 'w') as f2:
with open(f'{api_dir}/{cls}.md', 'w') as f2:
f2.write(f'# {module_name}.{cls}\n')
f2.write('```{eval-rst}\n')
f2.write(f'.. autoclass:: {module_name}.{cls}\n')
@ -38,7 +39,7 @@ def write_classes(f: TextIOWrapper, patterns: list[str], module_name: str, title
f2.write('```\n\n')
def write_functions(f: TextIOWrapper, patterns: list[str], module_name: str, title: str, description: str = '', exclude: list[str] = [], path_patterns: list[str] = ['*']) -> None:
def write_functions(f: TextIOWrapper, patterns: list[str], module_name: str, title: str, description: str = '', exclude: list[str] = [], path_patterns: list[str] = ['*'], api_dir: str = 'api') -> None:
"""Write the classes to the file."""
module = importlib.import_module(module_name)
@ -57,7 +58,7 @@ def write_functions(f: TextIOWrapper, patterns: list[str], module_name: str, tit
for func in functions:
if not func.startswith('_'):
with open(f'docs/source/api/{func}.md', 'w') as f2:
with open(f'{api_dir}/{func}.md', 'w') as f2:
f2.write(f'# {module_name}.{func}\n')
f2.write('```{eval-rst}\n')
f2.write(f'.. autofunction:: {module_name}.{func}\n')
@ -76,26 +77,32 @@ def write_dochtree(f: TextIOWrapper, title: str, items: list[str]):
if __name__ == "__main__":
# Ensure the output directory exists
os.makedirs('docs/source/api', exist_ok=True)
parser = argparse.ArgumentParser(description='Generate class and function documentation')
parser.add_argument('--api-dir', type=str, default='docs/source/api', help='Output directory for API documentation (default: api)')
args = parser.parse_args()
with open('docs/source/api/index.md', 'w') as f:
api_dir = args.api_dir
# Ensure the output directory exists
os.makedirs(api_dir, exist_ok=True)
with open(f'{api_dir}/index.md', 'w') as f:
f.write('# User API\n\n')
write_classes(f, ['*'], 'copapy', title='Classes')
write_classes(f, ['*'], 'copapy', title='Classes', api_dir=api_dir)
write_functions(f, ['*'], 'copapy', title='General functions', path_patterns=['*_autograd.py', '*_basic_types.py', '*_target.py'])
write_functions(f, ['*'], 'copapy', title='General functions', path_patterns=['*_autograd.py', '*_basic_types.py', '*_target.py'], api_dir=api_dir)
write_functions(f, ['*'], 'copapy', title='Math functions', path_patterns=['*_math*'], exclude=['get_42'])
write_functions(f, ['*'], 'copapy', title='Math functions', path_patterns=['*_math*'], exclude=['get_42'], api_dir=api_dir)
write_functions(f, ['*'], 'copapy', title='Vector functions', path_patterns=['*_vectors*'])
write_functions(f, ['*'], 'copapy', title='Vector functions', path_patterns=['*_vectors*'], api_dir=api_dir)
write_functions(f, ['*'], 'copapy', title='Matrix functions', path_patterns=['*_matrices*'])
write_functions(f, ['*'], 'copapy', title='Matrix functions', path_patterns=['*_matrices*'], api_dir=api_dir)
write_manual(f, ['NumLike'], title='Types')
#write_manual(f, ['NumLike'], title='Types')
with open('docs/source/api/backend.md', 'w') as f:
with open(f'{api_dir}/backend.md', 'w') as f:
f.write('# Backend\n\n')
write_classes(f, ['*'], 'copapy.backend', title='Classes')
write_classes(f, ['*'], 'copapy.backend', title='Classes', api_dir=api_dir)
write_functions(f, ['*'], 'copapy.backend', title='Functions')
write_functions(f, ['*'], 'copapy.backend', title='Functions', api_dir=api_dir)

View File

@ -7,5 +7,5 @@ api/backend
repo
```
```{include} start.md
```{include} ../build/start.md
```

View File

@ -0,0 +1,3 @@
# Stencil overview
```{include} ../build/stencils.md
```

142
docs/source/stencil_doc.py Normal file
View File

@ -0,0 +1,142 @@
import re
from pathlib import Path
import glob
import argparse
def extract_c_functions(stencils_path: str) -> dict[str, str]:
"""
Extract all C function names and their code from a stencils.c file.
Args:
stencils_path: Path to the stencils.c file
Returns:
Dictionary mapping function names to their complete code
"""
with open(stencils_path, 'r') as f:
content = f.read()
# Regex pattern to match C functions
# Matches: return_type function_name(parameters) { ... }
pattern = r'((?:STENCIL\s+extern|void|int|float|double)\s+\w+\s*\([^)]*\)\s*\{(?:[^{}]|\{[^{}]*\})*\})'
functions: dict[str, str] = {}
# Find all function matches
for match in re.finditer(pattern, content, re.MULTILINE | re.DOTALL):
func_code = match.group(1).strip()
# Extract function name using a simpler regex on the matched code
name_match = re.search(r'(?:STENCIL\s+extern)?\s*(?:void|int|float|double)?\s*(\w+)\s*\(', func_code)
if name_match:
func_name = name_match.group(1)
functions[func_name] = func_code
return functions
def extract_asm_section(asm_path: str) -> dict[str, str]:
"""
Extract assembly functions organized by section.
Args:
asm_path: Path to the stencils.asm file
Returns:
Dictionary with sections as keys, containing function dictionaries
"""
with open(asm_path, 'r') as f:
content = f.read()
# Split by "Disassembly of section"
sections = re.split(r'^Disassembly of section (.+?):', content, flags=re.MULTILINE)
result: dict[str, str] = {}
# Process sections (skip first empty element)
for i in range(1, len(sections), 2):
section_name = sections[i].strip()
section_content = sections[i + 1] if i + 1 < len(sections) else ""
if section_content:
result[section_name] = section_content.strip()
return result
def build_asm_code_dict(asm_glob_pattern: str) -> dict[str, dict[str, str]]:
"""
Build a dictionary of assembly code for all available architectures.
Args:
asm_glob_pattern: Glob pattern to find stencils.asm files
Returns:
Dictionary mapping architecture names to their asm_code dictionaries
"""
asm_code: dict[str, dict[str, str]] = {}
asm_files = glob.glob(asm_glob_pattern)
for asm_file in asm_files:
arch_name = Path(asm_file).parent.name.replace('runner-linux-', '')
try:
asm_code[arch_name] = extract_asm_section(asm_file)
print(f"Loaded assembly for {arch_name}")
except FileNotFoundError:
print(f"Warning: Assembly file not found for {arch_name}: {asm_file}")
return asm_code
# Example usage:
if __name__ == "__main__":
parser = argparse.ArgumentParser(description="Generate stencils documentation from C and assembly code")
parser.add_argument('--input', default='build/stencils.c', help='Path to input C file')
parser.add_argument('--asm-pattern', default='build/tmp/runner-*/stencils.asm', help='Glob pattern for assembly files')
parser.add_argument('--output', default='docs/build/stencils.md', help='Output markdown file path')
args = parser.parse_args()
# Get all C functions
functions = extract_c_functions(args.input)
# Build assembly code dictionary for all architectures
asm_code = build_asm_code_dict(args.asm_pattern)
#@norm_indent
def get_stencil_section(func_name: str) -> str:
c_code = functions[func_name]
section_name = '.text.' + func_name
arch_asm_code = ''
for arch in sorted(asm_code.keys()):
if section_name in asm_code[arch]:
arch_asm_code += f"""
### {arch}
```nasm
{asm_code[arch][section_name]}
```
"""
else:
arch_asm_code += f"\n### {arch}\nNo assembly found for this architecture\n"
return f"""
## {func_name}
```c
{c_code}
```
{arch_asm_code}
"""
md_code: str = ''
for function_name, code in functions.items():
md_code += get_stencil_section(function_name)
with open(args.output, 'wt') as f:
f.write(md_code)
print(f"Generated {args.output} with {len(functions)} stencil functions")

View File

@ -1,31 +1,16 @@
from copapy import value
from copapy.backend import Write, compile_to_dag, stencil_db_from_package
from copapy._binwrite import Command
import copapy as cp
input = value(9.0)
def compile_example(arch: str = 'native') -> None:
"""Test compilation of a simple program for x86_64."""
c1 = value(9.0)
#ret = [c1 / 4, c1 / -4, c1 // 4, c1 // -4, (c1 * -1) // 4]
ret = [c1 // 3.3 + 5]
#ret = [cp.sqrt(c1)]
#c2 = cp._math.get_42()
#ret = [c2]
out = [Write(r) for r in ret]
result = input ** 2 / 3.3 + 5
arch = 'native'
sdb = stencil_db_from_package(arch)
dw, _ = compile_to_dag(out, sdb)
dw, _ = compile_to_dag([Write(result)], sdb)
# Instruct runner to dump patched code to a file:
dw.write_com(Command.DUMP_CODE)
#print('* Data to runner:')
#dw.print()
dw.to_file('build/runner/test.copapy')
if __name__ == "__main__":
compile_example()