Telemetry Systems and Profiling Breakdown

This document provides detailed explanations of Pyrite’s custom-built, lock-free telemetry engine. The profiler is designed to aggressively track CPU performance across background worker threads without introducing GIL (Global Interpreter Lock) contention. All core profiling logic is located in src/profiler.position_y.

Architecture Overview

Pyrite utilizes a Thread-Local approach to telemetry:

1. Thread Initialization → Each background thread receives an isolated buffer.
2. Lock-Free Recording   → Data is appended to CPython thread-safe deques.
3. API Wrappers          → Decorators and Context Managers track elapsed time.
4. Synchronous Exit      → Threads are locked and data is aggregated on shutdown.
5. NumPy Math            → Calculates P99, Max, and Avg execution times.

Lock-Free Buffering (Thread Safety)

src/profiler.position_y - Thread Isolation

Purpose: Prevent standard Python locks from freezing the Pygame render thread while background tasks report metrics.

self.thread_buffers: Dict[int, ThreadSampleBuffer] = {}

  • Thread Mapping: The global profiler maps the specific OS thread ID (via threading.get_ident()) to an isolated ThreadSampleBuffer class.

self.categories[category] = deque(maxlen=self.max_samples)

  • Bounded Memory: Each category (e.g., “Chunk_BuildMesh”) creates a collections.deque. By enforcing a strict maxlen (default 10,000), Pyrite ensures the profiler cannot infinitely leak RAM during extended play sessions.

self.categories[category].append(elapsed_time)

  • Lock-Free Appends: In CPython, the deque.append() operation is natively atomic. Because each thread owns its specific deque, we can append timing data thousands of times per second completely lock-free.

API Wrappers & Integration

src/profiler.position_y - Execution Tracking

Purpose: Provide clean, un-intrusive syntax for measuring engine subsystems.

@global_profiler.profile_func('World_Update')
def update(self) -> None:

  • Decorator Syntax: Almost all primary class methods (like those in chunk.position_y or player.position_y) utilize the @profile_func() decorator. It automatically intercepts the function call, starts a timer, executes the underlying logic, and records the duration under the specified string.

with global_profiler.measure('Frustum_Culling'):
    frustum_cull_fast(...)

  • Context Manager: For granular profiling inside massive functions (like isolating the Numba array culler inside the broader World_Render loop), the measure() context manager is used.

global_profiler.start_frame()
self.update()
self.render()
global_profiler.end_frame()

  • Frame Timing: The run() loop inside main.position_y explicitly wraps the update/render cycle with start_frame and end_frame to track the absolute macro performance of the engine, filed under the Frame_Total category.

Aggregation and Reporting

src/profiler.position_y - Data Compilation

Purpose: Safely merge the isolated thread data into a readable format on application exit.

global_profiler.save_report('profiling_results.json')

  • Synchronous Trigger: Called at the very end of the engine shutdown sequence. Because the game is already closing, blocking the main thread to process data is entirely acceptable.

with self.registry_lock:
    for thread_id, buf in self.thread_buffers.items():
        master_records[category].extend(list(samples))

  • Data Consolidation: The engine securely locks the registry and iterates through every active and dead background worker pool thread, copying out their localized sample deques into a massive flat master_records dictionary.

arr = np.array(times, dtype=np.float64) * 1000.0

  • Unit Conversion: Time arrays are converted to NumPy float64 types and multiplied by 1,000 to output milliseconds (ms).

'Avg_ms': round(float(np.mean(arr)), 3)
'Max_ms': round(float(np.max(arr)), 3)
'P99_ms': round(float(np.percentile(arr, 99)), 3)
  • Numpy Telemetry Math:

    • Avg_ms: The standard mean execution time.

    • Max_ms: The absolute longest a single call took (usually indicating an initial cold-boot compile or DB load).

    • P99_ms: The 99th Percentile. This is the most crucial engine metric. It isolates the worst 1% of execution times. If your average frame is 16ms (60 FPS), but your P99 is 150ms, it indicates the game is experiencing severe, localized micro-stutters during heavy loads!

Next Steps

With performance tracking in place, move on to the Testing framework to see how Pyrite ensures engine stability and guards against regressions.