Performance Benchmark Results

Generated on 2026-09-20 06:03:28

This report compares the runtime performance, peak memory usage, and file size of equivalent programs implemented in:

• C
• Sere
• Python

Each implementation was executed 250 times.

Summary

Relative performance

Using C as a baseline:

• Sere averaged 0.47x the runtime of C.
• Python averaged 2.05x the runtime of C.
• Python averaged 4.37x the runtime of Sere.

A value close to 1.00x means the implementations performed similarly.

Runtime Performance

Runtime interpretation

Average represents the mean execution time across all benchmark runs.

Fastest is the shortest observed execution time.

Slowest is the longest observed execution time.

Median is the middle execution time and can be more resistant to occasional operating-system scheduling spikes.

Std Dev shows how much timing variation occurred between runs. A lower standard deviation generally indicates more consistent performance.

Memory Usage

Memory measurements represent the process Resident Set Size, or RSS, observed while the program was running.

RSS approximates the amount of physical memory currently mapped into the process.

The benchmark polls memory approximately once every millisecond, so extremely short-lived memory peaks may not always be observed.

File Size

For C and Sere, this represents the compiled executable size.

For Python, this represents the source file size rather than the size of the Python interpreter or its runtime environment.

Because of this, Python's value is not directly equivalent to the compiled executable sizes.

Individual Runtime Results

These are the raw runtime measurements for every benchmark iteration.

They are useful for identifying outliers, scheduler interference, startup noise, or unusually inconsistent runs.

Benchmark Methodology

Each program is launched as a new operating-system process using Python's subprocess.Popen.

Execution time is measured using time.perf_counter().

The timer begins immediately before process creation and ends after the process has terminated and its output has been collected.

Memory consumption is monitored with psutil using process.memory_info().rss.

The process is sampled approximately every 1 millisecond.

Important Benchmarking Context

These results measure more than just the execution speed of the program itself.

Because a new process is started for every benchmark run, the measured time also includes some operating-system overhead such as:

• process creation
• executable loading
• dynamic library loading
• runtime initialization
• Python interpreter startup
• process teardown
• stdout and stderr pipe handling

For programs that execute extremely quickly, these costs can represent a significant percentage of total measured runtime.

This is especially important when comparing compiled languages such as C and Sere.

If the actual benchmark workload only takes a fraction of a millisecond, Windows process startup may dominate the result.

Python Comparison Caveat

Python has a larger startup cost because each benchmark run launches a complete Python interpreter.

This benchmark therefore measures:

Python interpreter startup + script execution

rather than only the execution time of the Python benchmark workload itself.

This is still useful when evaluating short command-line programs, but it should not be interpreted as a pure comparison of language execution speed.

For CPU-bound language benchmarks, a better test is usually to perform the benchmark workload many thousands or millions of times inside each process.

Memory Measurement Caveat

Peak memory is obtained through polling.

Because memory is sampled every millisecond, a memory allocation that exists for less than one polling interval may not be recorded.

The reported peak should therefore be considered an observed peak rather than a guaranteed absolute maximum.

Interpreting Sere vs C

C provides a useful baseline because it compiles directly to native machine code with minimal runtime overhead.

Sere is also compiled, so comparing it to C can help reveal overhead introduced by areas such as:

• generated LLVM IR
• runtime initialization
• garbage collection
• bounds checks
• abstraction layers
• standard library initialization
• allocator behavior
• code generation quality

A Sere result close to C suggests that the generated native code and runtime overhead are relatively small for this workload.

Larger differences should be investigated using larger workloads before drawing conclusions, because startup overhead can heavily distort very short benchmarks.

Recommendations for More Reliable Results

For more serious compiler benchmarking, consider:

1. Increasing the number of runs.
2. Running benchmark workloads many times inside each executable.
3. Performing several warm-up runs before collecting measurements.
4. Testing multiple workloads instead of a single program.
5. Measuring CPU-heavy, allocation-heavy, and I/O-heavy programs separately.
6. Building C and Sere with equivalent optimization levels.
7. Closing unnecessary background applications.
8. Running benchmarks while the system is idle.
9. Comparing generated assembly or LLVM IR for suspicious performance differences.
10. Reporting median and variance alongside average runtime.

For example:

will usually provide a more stable average than a small number of runs.

Benchmark Environment

This report was generated automatically by perf.py.

Python executable:

Platform:

Benchmark files:

C:      C:\Users\jackw\OneDrive\Desktop\git-projects\sere\test-c.exe
Sere:   C:\Users\jackw\OneDrive\Desktop\git-projects\sere\test.exe
Python: C:\Users\jackw\OneDrive\Desktop\git-projects\sere\test.py

Generated automatically by the Sere benchmark script.