Skip to content

LLM Context Guide: Understanding pyegp-parser JSON Output

This document is designed to be included as context for Large Language Models (LLMs) that need to consume, analyze, or answer questions about the JSON output produced by the pyegp-parser. It provides the semantic understanding that raw JSON alone cannot convey.

What is an EGP File?

An .egp file is a SAS Enterprise Guide project — a binary ZIP archive used by the SAS Enterprise Guide desktop application (versions 7.x–8.x). It contains:

  • project.xml — the master manifest describing all project elements, their metadata, relationships, data connections, query definitions, visual layout, and execution configuration
  • Task config XML files — wizard-generated configurations for built-in tasks (imports, exports, EG tasks)
  • SAS code files — user-written or auto-generated .sas code
  • Execution logs — output from previous SAS session runs
  • ODS results — rendered output files (HTML, PDF, PowerPoint, Excel)

The pyegp-parser extracts all of this content into a single structured project.json file.

What Does the JSON Represent?

Each project.json is a complete, structured representation of one SAS Enterprise Guide project. Think of it as a fully decompiled project — everything a SAS developer configured in the GUI is captured here in machine-readable form.

A typical project contains: - Data pipeline logic: What datasets are read, how they're joined/filtered/transformed, and where results are written - SAS code: The actual SAS programs (PROC SQL, DATA steps, macro calls) that execute the transformations - Execution graph: A directed acyclic graph (DAG) showing which tasks depend on which others - Connection metadata: What SAS servers, libraries, and tables are referenced - Visual layout: Where nodes are positioned in the process flow diagram

Key Concepts for Interpretation

Element IDs and Cross-References

Every object in an EGP project has a unique ID (e.g., "CodeTask-a1b2c3d4e5f6"). These IDs are used to create relationships:

  • input_ids on an element points to its upstream dependencies
  • parent_id on a Code/Log element points to the task that owns it
  • container on an element indicates which Process Flow it belongs to
  • shortcut.parent_id points to a DataItem or ExternalFileItem
  • DAG connections use source_id/target_id to define execution order

When navigating the JSON, use these IDs to trace data lineage and execution flow.

The _type Discriminator

Every nested object has a _type field indicating its class. This allows you to determine what kind of object you're looking at without checking which array it came from:

  • "ParsedProject" — the root
  • "ElementMetadata" — basic metadata for any element
  • "CodeElement" — a structured code block
  • "CodeTaskElement" — a user-written code task
  • "ImportTaskElement" — a data import wizard task
  • "QueryModel" — a query builder definition
  • "DAGModel" — an execution dependency graph
  • "DataItem" — a dataset reference
  • "ShortCutToData" / "ShortCutToFile" — indirect references
  • "ProcessFlowContainer" — a grouping of tasks with its DAG
  • "ExternalObject" — a reference to a file/library outside the project

Element Type Hierarchy

The type field on ElementMetadata is a fully-qualified .NET-style type string. The final segment tells you the element category:

Type suffix Meaning
Project The root project element
ProcessFlowContainer A named process flow (tab in the GUI)
Query A Query Builder query (SQL via GUI)
CodeTask A user-written SAS code task
ImportTask A data import wizard task
EGTask A built-in Enterprise Guide wizard task
ExportTask A data export wizard task
AppendTask A data append/concatenation task
ShortCutToData A reference linking to a DataList item
ShortCutToFile A reference linking to an ExternalFileList item
Data A dataset registered in the DataList
ExternalFile An external file registered in ExternalFileList
Log A log output display element
Code A structured code element (auto-generated by tasks)
ProjectLog The project-level execution log

Section-by-Section Interpretation Guide

source

Provenance metadata. Tells you where the file came from and when it was parsed.

source.file_name → original .egp filename
source.file_size_bytes → how large the archive was
source.total_zip_entries → how many files were inside the archive

metadata

Project identity. The label is the human-readable project name. The eg_version tells you which Enterprise Guide version created it (affects XML format).

settings

Execution configuration. Key fields: - submit_to_grid — whether code runs on a grid computing cluster - action_on_error — what happens when a task errors ("StopProcessFlow", "Continue", etc.) - use_relative_paths — whether file references are relative or absolute

parameters

Project-level macro variables (name/value pairs) that are set before any task executes.

elements

All elements in document order — this is a flat list of ElementMetadata for every element in the project. Use this for a complete inventory. Each entry has id, label, type, container, and input_ids.

The input_ids array is crucial: it tells you which elements feed into this one. This is the data lineage at the element level.

containers

Process Flow Containers with their execution DAGs. Each container has: - metadata — the container's name and ID - dag.nodes — element IDs in topological execution order (first to last) - dag.connections — directed edges with source_id → target_id and a resource_dependency flag

The DAG is the execution plan. Nodes are executed in the order listed. Connections show which tasks must complete before others can start. A resource_dependency: true connection means the target needs a dataset produced by the source.

queries

Query Builder elements. Each entry has: - metadata — identifies the query element - submitable — execution settings (which server to run on) - query_model — the full query definition: - input_tables — source tables with server/library/member details - result_items — output columns (which columns to SELECT) - calculations — computed columns (expressions/formulas) - join_items — JOIN conditions between tables - where_filters — WHERE clause filters - having_filters — HAVING clause filters (for aggregates) - order_items — ORDER BY specification - group_items — GROUP BY specification

This is equivalent to a SQL query defined via the visual Query Builder. You can reconstruct the SQL from these fields.

tasks

All task-type elements. The _type field tells you which kind:

  • CodeTaskElement — contains code_content (the SAS program text) and submitable (server config). This is the most common task type in real projects.
  • ImportTaskElement — a data import wizard; task_config contains the full import configuration XML.
  • EGTaskElement — a built-in wizard task (e.g., Summary Statistics, Linear Regression); eg_task_clsid identifies which wizard, task_config has the configuration.
  • ExportTaskElement — outputs data to a file.
  • AppendTaskElement — concatenates multiple datasets.

code_elements

Structured code blocks associated with tasks. When Enterprise Guide generates code for a task, it wraps it in a Code element with sections:

  • macro_assign_code — macro variable assignments run BEFORE the task (sets _CLIENTTASKLABEL, project path, etc.)
  • begin_app_code — ODS setup and application preamble
  • begin_user_code — user code that runs before the main task
  • task_code — the main SAS code (this is what the task actually does)
  • end_user_code — user code after the main task
  • end_app_code — ODS closing and cleanup
  • macro_unassign_code — macro cleanup run AFTER the task

For understanding what a task does, focus on task_code. The other sections are Enterprise Guide boilerplate.

The parent_id links back to the task that owns this code. The input_ids in metadata shows what feeds into it.

shortcuts

Indirect references connecting process flow tasks to data items. A ShortCut doesn't contain data itself — it points to a DataItem or ExternalFileItem via parent_id.

Use shortcuts to understand which data a task reads or writes. Follow parent_id to find the actual dataset in data_list or external_files.

data_list

Registered datasets in the project. Each has: - element — metadata (label is typically the table name) - data_model — connection details: - server — SAS server name - table — dataset/table name - member_type — "DATA" or "VIEW" - decoded_dna — full decoded connection descriptor with hierarchy information

external_files

External file references (CSV, Excel, text files). Similar to data_list but for non-SAS files. The file_type_type tells you the format ("CSV", "XLSX", etc.).

external_objects

Objects referenced but not embedded in the project (external libraries, database connections). Each has name, type, path, and additional metadata.

log_elements

Log display configuration (page size, line size, orientation). These control how SAS log output is rendered. The parent_id links to the task whose log output this configures.

visual_layout

Node positioning for the process flow diagram. Each TaskGraphic has: - element — references an element ID - pos_x, pos_y — pixel position - width, height — node size

This is purely visual metadata — useful if you need to reconstruct or render the process flow diagram.

binary_entries

ODS output files (PowerPoint, Excel, HTML, PDF) produced by task execution. These are tracked by path and size but their binary content is not extracted into JSON. The task_id links to the producing task.

completeness_summary and unprocessed_entries

Quality metrics: - total_entries — how many files were in the ZIP - processed_entries — how many the parser understood - unprocessed_entries — leftovers (usually benign: temp files, cached data)

If completeness_warning is true, check unprocessed_entries to see what was skipped.

Common Analysis Tasks

"What does this project do?"

  1. Read metadata.label for the project name
  2. Look at containers[0].dag.nodes for the execution order
  3. For each node ID, find it in tasks or queries to see what it does
  4. Read code_elements where parent_id matches a task to get the SAS code

"What data does this project read?"

  1. Look at data_list for all registered datasets
  2. Check external_files for non-SAS data sources
  3. For each, examine data_model.server and data_model.table (or file_type_type for files)

"What is the data lineage / execution order?"

  1. Find the main container in containers
  2. Read dag.nodes — this is the topological execution order
  3. Read dag.connections — these are the dependency edges
  4. Use input_ids on each element to trace what feeds into what

"Show me the SQL / SAS code"

  1. For Query Builder queries: reconstruct from queries[].query_model (input_tables + result_items + joins + filters)
  2. For Code tasks: read tasks[].code_content where _type is CodeTaskElement
  3. For auto-generated code: read code_elements[].task_code

"What servers and libraries are used?"

  1. Scan data_list[].data_model.server for SAS servers
  2. Scan queries[].submitable.server for query execution servers
  3. Scan tasks[].submitable.server for task execution servers
  4. Check external_files[].decoded_dna for file system paths

Timestamps

EGP files store timestamps as Windows FILETIME ticks (100-nanosecond intervals since January 1, 1601). To convert to a human-readable date:

ticks = 638396640000000000
epoch_diff = 621355968000000000  # difference between 1601 and 1970 in ticks
unix_timestamp = (ticks - epoch_diff) / 10_000_000
# → 2024-01-01T00:00:00 (UTC)

Null Values

Fields set to null mean the information was not present in the source XML. This is normal — not every project uses every feature. For example: - project_log: null — the project doesn't have a ProjectLog element - code_content: null — a CodeTask that hasn't been executed yet (no code file in archive) - task_config: null — a task whose config file is missing from the archive

ID Format Convention

Element IDs follow the pattern {TypePrefix}-{RandomAlphanumeric}: - CodeTask-a1b2c3d4e5f6 - Query-b2c3d4e5f6a7 - ImportTask-c3d4e5f6a7b8 - ProcessFlowContainer-d4e5f6a7b8c9 - Data-abc123def456

The prefix indicates the element type. The suffix is a random identifier unique within the project.

Relationship Map

ParsedProject
├── containers[] ─── ProcessFlowContainer
│   └── dag ──────── DAGModel
│       ├── nodes[] ─────── (element IDs in execution order)
│       └── connections[] ── Connection (source_id → target_id)
│
├── elements[] ───── ElementMetadata
│   ├── id ──────────── (unique identifier, referenced everywhere)
│   ├── container ───── (points to → ProcessFlowContainer.id)
│   └── input_ids[] ─── (points to → other element IDs)
│
├── queries[] ────── {metadata, submitable, query_model}
│   └── query_model.input_tables[].data_id ── (points to → DataItem.id)
│
├── tasks[] ──────── CodeTaskElement / ImportTaskElement / ...
│   └── metadata.id ── (referenced by code_elements[].parent_id)
│
├── code_elements[] ── CodeElement
│   ├── parent_id ───── (points to → task.metadata.id)
│   └── metadata.input_ids[] ── (points to → element IDs)
│
├── shortcuts[] ───── ShortCutToData / ShortCutToFile
│   └── parent_id ───── (points to → data_list[].element.id
│                          or external_files[].element.id)
│
├── data_list[] ───── DataItem
│   ├── element.id ──── (referenced by shortcuts)
│   └── shortcut_list[] ── (ShortCut IDs that reference this item)
│
├── external_files[] ── ExternalFileItem
│   ├── element.id ──── (referenced by shortcuts)
│   └── shortcut_list[] ── (ShortCut IDs that reference this item)
│
└── log_elements[] ── LogElement
    └── parent_id ───── (points to → task or element that owns this log)

Summary

When consuming this JSON: 1. Start with metadata and source to understand what project this is 2. Use containers[].dag to understand execution flow 3. Use tasks and code_elements to understand what each step does 4. Use data_list, external_files, and shortcuts to understand data lineage 5. Use queries to understand GUI-defined SQL logic 6. Cross-reference using element IDs — they are the glue connecting everything