Optimized Execution for Semantic Operators

Palimpzest (PZ) enables developers to process unstructured data (i.e. documents, images, audio, etc.) using semantic operators -- i.e. AI-powered data transformations. PZ programs are declarative, which means they express what computation should be performed without specifying exactly how to perform it. This allows PZ's optimizer to select the best way to execute each semantic operator in order to minimize cost, minimize latency, or maximize quality, possibly subject to constraints on the other dimensions.

✉️ Example: Processing Emails with Semantic Operators

Download Example Dataset

The following code snippet sets up PZ and downloads a small datast of emails:

# install palimpzest with pip
$ pip install palimpzest

# you can also install palimpzest with uv for a faster install
# $ uv pip install palimpzest

# set OpenAI API key
$ export OPENAI_API_KEY="<your-api-key>"

# download and extract emails
$ wget https://palimpzest-workloads.s3.us-east-1.amazonaws.com/emails.zip
$ unzip emails.zip

The following PZ program extracts the subject, sender, and summary of emails which contain first-hand discussion of specific business transactions. The program is written in a high-level declarative language, and PZ automatically optimizes the execution of each semantic operator to maximize output quality:

import palimpzest as pz

# load the emails into a dataset
emails = pz.TextFileDataset(id="enron-emails", path="emails/")

# filter for emails matching natural language criteria
emails = emails.sem_filter(
    'The email refers to one of the following business transactions: "Raptor", "Deathstar", "Chewco", and/or "Fat Boy")',
)
emails = emails.sem_filter(
    "The email contains a first-hand discussion of the business transaction",
)

# extract structured fields for each email
emails = emails.sem_map([
    {"name": "subject", "type": str, "desc": "the subject of the email"},
    {"name": "sender", "type": str, "desc": "the email address of the sender"},
    {"name": "summary", "type": str, "desc": "a brief summary of the email"},
])

# execute the program and print the output
output = emails.run(max_quality=True)

print(output.to_df(cols=["filename", "sender", "subject", "summary"]))

The output from this program is shown below:

                               filename                                    subject                    sender                                            summary
0  whalley-g-merchant-investments-3.txt         Enron Principal Investments Update   kevin.garland@enron.com  Kevin Garland provides an update on Enron Prin...
1              kaminski-v-inbox-291.txt  RE: Pricing of restriction on Enron stock           baker@enron.com  Ron Baker clarifies to Vince Kaminski that the...
2               delainey-d-sent-683.txt                                 Re: Raptor  david.delainey@enron.com  David Delainey responds about hedging and the ...
3               kaminski-v-inbox-92.txt                                FW: Raptors      j.kaminski@enron.com  Vince Kaminski forwards a previously sent mess...
4               delainey-d-sent-295.txt                                   AIG Fund  david.delainey@enron.com  David Delainey advises colleagues to revise DA...

There are a few features of this program which are worth highlighting:

1. The program creates a dataset from the directory of emails and defines a series of semantic operations on that dataset:
   • sem_filter() selects for emails which satisfy each natural language predicate
   • sem_map() specifies a set of fields which PZ must compute
2. The user does not specify how each operation is performed -- they simply declare what they want PZ to compute
   • This is what makes PZ declarative
3. Internally, PZ's optimizer determines the best way to execute each semantic operator
   • In this example, PZ optimizes for output quality because the user sets max_quality=True
4. The output is not generated until the call to emails.run()
   • i.e. PZ uses lazy evaluation

🛠️ Naive Optimization

The program above is optimized using naive prior beliefs about the quality of different operator implementations for sem_filter and sem_map. In brief, PZ will implement each sem_filter and sem_map operation using the available LLM with the highest score on MMLU-Pro. (To minimize cost (or latency) instead, we could call emails.run(min_cost=True) (or min_time=True) and PZ would use the cheapest (or fastest) available LLM based on per-token costs (or latencies)).

In order to leverage the full power of PZ's optimizer, we need to provide PZ with a pz.Validator which can evaluate the quality, cost, and latency of different operator implementations for each semantic operation.

✨ Optimizing Execution with a Validator

The real power of Palimpzest comes from its ability to test multiple implementations of each operator and select the best one based on empirical performance. For example, PZ can select which model to use for each semantic operation -- perhaps opting for a cheaper model that performs well on a given semantic filter or a more expensive model for a challenging semantic map. However, PZ's space of optimizations extends beyond model selection (also called "model routing") to include:

• Model selection
• Ensemble methods (e.g., Mixture-of-Agents)
• Refinement strategies (e.g., using a model to propose an answer, a second model to critique the answer, and a third model to refine the answer)
• Context reduction strategies (e.g., using embedding similarity to feed only the top-k most relevant chunks from the input context into the LLM)
• And more!

In order to make use of these advanced optimizations, we need to modify the above program to use a pz.Validator with .optimize_and_run():

import palimpzest as pz

...

# execute the program and print the output
validator = pz.Validator(model=pz.Model.GPT_5)
output = emails.optimize_and_run(max_quality=True, validator=validator)

print(output.to_df(cols=["filename", "sender", "subject", "summary"]))

The call to .optimize_and_run() will cause PZ to first perform an optimization loop where it samples different implementations of each semantic operator and evaluates them using the pz.Validator (in this case, GPT-5). After the optimization loop, PZ will select the best implementation of each operator and then execute the optimized program.

The output from this optimized program is shown below:

                                filename                                      subject                        sender                                            summary
0                delainey-d-sent-295.txt                                     AIG Fund      david.delainey@enron.com  David Delainey advises toning down claims of s...
1      kaminski-v-all-documents-2352.txt                         RE: Cross-Guarantees           ron.baker@enron.com  Ron Baker sends the latest drafts of three Cro...
2                delainey-d-sent-318.txt  Re: ENA Comp suggestions for Project Raptor      david.delainey@enron.com  David Delainey tells David Oxley to drop the c...
3             giron-d-sent-items-200.txt                           FW: Enron Mentions            c..giron@enron.com  Forwarded email compiling numerous news articl...
4                delainey-d-sent-683.txt                                   Re: Raptor      david.delainey@enron.com  David Delainey discusses hedging and restructu...
5   whalley-g-merchant-investments-3.txt           Enron Principal Investments Update       kevin.garland@enron.com  Update on Enron Principal Investments covering...
6    parks-j-deleted-items-913-short.txt                                   FW: E memo  gregory.schockling@enron.com  Gregory Schockling forwards an email chain reg...
7      kaminski-v-all-documents-2355.txt         Raptor Position Reports for 12/28/00           ron.baker@enron.com  Sends the latest Daily Position Report files f...
8               kaminski-v-inbox-291.txt    RE: Pricing of restriction on Enron stock               baker@enron.com  Ron Baker clarifies confusion about parts of a...
9                kaminski-v-inbox-92.txt                                  FW: Raptors          j.kaminski@enron.com  Vince Kaminski forwards a previously sent mess...
10            beck-s-notes-inbox-166.txt              Re: Enron Raptor I P&L Reversal        shona.wilson@enron.com  Shona Wilson informs recipients that the DPR c...

The optimized output contains more relevant emails than the naively optimized program, illustrating the benefit of PZ's optimization framework. In our user guide on optimization, we also show how to use labeled data (instead of an LLM) to drive optimization which can further improve performance.

📈 Declarative Optimization for AI

The core philosophy behind PZ is that programmers should simply specify the high-level logic of their AI programs while offloading much of the performance tuning to a powerful optimizer. Of course, users should still be able to fully control their program and override / assist the optimizer (if needed) to get the best possible performance.

🚀 More Semantic Operators

This email processing example only showcases a small set of the semantic operators implemented in PZ. Other operators include:

• sem_join() which performs a semantic join between two datasets
• sem_aggregate() which performs a semantic aggregation over a dataset
• sem_flat_map() which performs semantic map operations producing multiple outputs per input (e.g. extracting all the authors on a single paper)
• sem_topk() which takes a vector database and a search string as input and retrieves the top-k relevant entries from the database
• map(), flat_map(), filter(), and join() which are the relational equivalents of sem_map(), sem_flat_map(), sem_filter(), and sem_join()
• groupby(), count(), average(), limit(), and project() which mirror their implementations in frameworks like Pandas and Spark.

🙋🏽 Join our community

We strongly encourage you to join our Discord server where we are happy to help you get started with PZ.

➡️ What's Next?

The rest of our Getting Started section will:

1. Help you install PZ
2. Explore more of PZ's features in our Quick Start Tutorial
3. Give you an overview of our User Guides which discuss features of PZ in more depth