{ "cells": [ { "cell_type": "markdown", "id": "93e1ab4b3d759339", "metadata": { "collapsed": false }, "source": [ "## Data annotation\n", "\n", "For structured data extraction, it is indeed essential to define a structure into which the data is supposed to be stored. This process of defining and applying structure to data is often referred to as data annotation. The choice of data structure and format is crucial and can significantly impact the efficiency and effectiveness of your data extraction process.\n", "\n", "### Considerations for choosing a data structure\n", "While there's no one-size-fits-all answer, here are some key points to consider when deciding on a data structure:\n", "\n", "- Nested vs. flat: Certain formats like YAML or JSON are better at expressing dependencies and hierarchies compared to flat formats like CSV.\n", "- Verbosity and human readability: Some formats (e.g., XML) can be more verbose, which may affect readability.\n", "- Type annotations and documentation: It's best practice to include as much metadata as possible, including data types and descriptions.\n", "- Machine readability: Consider how easily the format can be parsed by various programming languages and tools.\n", "- Extensibility: Choose a format that allows for easy addition of new fields or structures as your data needs evolve.\n", "- Standardization: Consider if there are any industry-standard formats for your specific domain.\n", "\n", "```{admonition} Old formats regain attention \n", ":class: note \n", "\n", "

I'm switching from team JSON to team XML for LLM prompts. Escaping JSON and writing JSON is much more limiting. You can just yeet XML tags wherever and not have to worry about escaping/format/validity.

— Andrew White 🐦‍⬛/acc (@andrewwhite01) May 2, 2024
\n", "\n", "Note also that models have different preferences for formats. [For example, Claude was reported to perform very well with XML, whereas GPT seems to prefer JSON](https://www.oreilly.com/radar/what-we-learned-from-a-year-of-building-with-llms-part-i/).\n", "\n", "```\n", "\n", "```{admonition} Examples of various serialization formats\n", ":class: dropdown \n", "\n", "\n", " For modeling reactions, we can serialize data in a simple data model in different formats. \n", "\n", " Description of our data model:\n", "\n", " - A reaction has a name, reactants, products, and conditions.\n", " - Each reactant and product has a chemical formula and amount.\n", " - Conditions include temperature and pressure.\n", "\n", " #### CSV example\n", "\n", " ```\n", " reaction_name,reactant_formula,reactant_amount,product_formula,product_amount,temperature,pressure\n", " Combustion of Methane,CH4,1 mol,CO2,1 mol,298 K,1 atm\n", " Combustion of Methane,O2,2 mol,H2O,2 mol,298 K,1 atm\n", " Photosynthesis,6 CO2,6 mol,C6H12O6,1 mol,300 K,1 atm\n", " Photosynthesis,6 H2O,6 mol,O2,6 mol,300 K,1 atm\n", " ```\n", "\n", " #### YAML Example\n", "\n", " ```\n", " reactions:\n", " - name: Combustion of Methane\n", " reactants:\n", " - formula: CH4\n", " amount: 1 mol\n", " - formula: O2\n", " amount: 2 mol\n", " products:\n", " - formula: CO2\n", " amount: 1 mol\n", " - formula: H2O\n", " amount: 2 mol\n", " conditions:\n", " temperature: 298 K\n", " pressure: 1 atm\n", " - name: Photosynthesis\n", " reactants:\n", " - formula: 6 CO2\n", " amount: 6 mol\n", " - formula: 6 H2O\n", " amount: 6 mol\n", " products:\n", " - formula: C6H12O6\n", " amount: 1 mol\n", " - formula: O2\n", " amount: 6 mol\n", " conditions:\n", " temperature: 300 K\n", " pressure: 1 atm\n", " ```\n", "\n", " #### JSON Example\n", "\n", " ```\n", " {\n", " \"reactions\": [\n", " {\n", " \"name\": \"Combustion of Methane\",\n", " \"reactants\": [\n", " { \"formula\": \"CH4\", \"amount\": \"1 mol\" },\n", " { \"formula\": \"O2\", \"amount\": \"2 mol\" }\n", " ],\n", " \"products\": [\n", " { \"formula\": \"CO2\", \"amount\": \"1 mol\" },\n", " { \"formula\": \"H2O\", \"amount\": \"2 mol\" }\n", " ],\n", " \"conditions\": {\n", " \"temperature\": \"298 K\",\n", " \"pressure\": \"1 atm\"\n", " }\n", " },\n", " {\n", " \"name\": \"Photosynthesis\",\n", " \"reactants\": [\n", " { \"formula\": \"6 CO2\", \"amount\": \"6 mol\" },\n", " { \"formula\": \"6 H2O\", \"amount\": \"6 mol\" }\n", " ],\n", " \"products\": [\n", " { \"formula\": \"C6H12O6\", \"amount\": \"1 mol\" },\n", " { \"formula\": \"O2\", \"amount\": \"6 mol\" }\n", " ],\n", " \"conditions\": {\n", " \"temperature\": \"300 K\",\n", " \"pressure\": \"1 atm\"\n", " }\n", " }\n", " ]\n", " }\n", " ```\n", "\n", " #### XML example \n", "\n", " ```\n", " \n", " \n", " Combustion of Methane\n", " \n", " \n", " CH4\n", " 1 mol\n", " \n", " \n", " O2\n", " 2 mol\n", " \n", " \n", " \n", " \n", " CO2\n", " 1 mol\n", " \n", " \n", " H2O\n", " 2 mol\n", " \n", " \n", " \n", " 298 K\n", " 1 atm\n", " \n", " \n", " \n", " Photosynthesis\n", " \n", " \n", " 6 CO2\n", " 6 mol\n", " \n", " \n", " 6 H2O\n", " 6 mol\n", " \n", " \n", " \n", " \n", " C6H12O6\n", " 1 mol\n", " \n", " \n", " O2\n", " 6 mol\n", " \n", " \n", " \n", " 300 K\n", " 1 atm\n", " \n", " \n", " \n", " ```\n", "\n", "```\n", "\n", "### Additional aspects to consider\n", "#### Schema definition languages\n", "When working with complex data structures, it can be beneficial to use schema definition languages. These allow you to formally define your data structure, which can then be used for validation, documentation, and even code generation. Some examples include:\n", "\n", "- [JSON Schema](https://json-schema.org) for JSON.\n", "- [XSD (XML Schema Definition)](https://www.w3.org/TR/xmlschema-1/) for XML.\n", "- [LinkML (Linked data Modeling Language)](https://linkml.io) for various formats.\n", "\n", "\n", "```{admonition} LinkML\n", ":class: tip\n", "\n", "[LinkML (Linked data Modeling Language)](https://linkml.io) is a framework for modeling and working with structured data. It can be useful for structured data extraction using LLMs in several ways:\n", "\n", "- Schema definition: LinkML allows you to define schemas for your data models, specifying classes, attributes, and relationships. This provides a structured foundation for extracting information.\n", "- Interoperability: It promotes interoperability between different data formats and systems, making it easier to integrate extracted data into various workflows.\n", "- Validation: LinkML schemas can be used to validate extracted data, ensuring it conforms to the defined structure and constraints.\n", "- Code generation: It can generate code in various languages (e.g., Python, Java) based on your schema, facilitating data manipulation and processing.\n", "- Semantic web integration: LinkML is compatible with semantic web technologies, allowing extracted data to be easily linked and integrated with existing knowledge graphs.\n", "```\n", "\n", "#### Semantic annotations\n", "Beyond just structuring data, consider adding semantic annotations. This involves linking your data to standardized vocabularies or ontologies, which can greatly enhance the interoperability and machine-readability of your data. For example:\n", "\n", "- Using standard chemical identifiers (like InChI or SMILES) in your chemical formulas.\n", "- Linking temperature and pressure units to standard unit ontologies ([QUDT ontology](https://qudt.org)) or materials science ontologies ([EMMO](https://emmo-repo.github.io), [MatOnto](https://matportal.org/ontologies/MATONTO?p=classes)).\n", "\n", "#### Handling missing or uncertain data\n", "In real-world data extraction scenarios, you often encounter missing or uncertain data. Your annotation scheme should have a way to represent:\n", "\n", "- Missing data (e.g., null values, specific placeholders).\n", "- Uncertain data (e.g., confidence scores, possible value ranges).\n", "\n", "##### Versioning\n", "If your data model is likely to evolve over time, consider how you'll handle versioning. This might involve:\n", "\n", "- Including version numbers in your schema.\n", "- Maintaining backwards compatibility.\n", "- Documenting changes between versions.\n" ] }, { "cell_type": "markdown", "id": "f67d9499", "metadata": {}, "source": [ "```{admonition} Data model, schema format \n", "*Data Model:* Represents the structure and organization of data, defining how data is stored, organized, and manipulated. In this case, it includes reactions, reactants, products, and conditions.\n", "\n", "*Schema:* Defines the structure of data within a particular format, specifying the data types, constraints, and relationships. For example, an XML Schema (XSD) or JSON Schema can be used to validate the structure of XML or JSON data, respectively.\n", "\n", "*Format:* Refers to the way data is encoded and represented for storage or transmission. CSV, YAML, JSON, and XML are different formats that encode data in different way.\n", "```" ] }, { "cell_type": "markdown", "id": "34a0440831ddebf0", "metadata": { "collapsed": false }, "source": [ "In practice, it is often most convenient to define a data schema in code. This has multiple advantages: \n", "\n", "- the data schema can be tracked with all other code using version control, \n", "- there are existing routines for export in various formats,\n", "- data can be conveniently accessed in code, e.g., via class attributes." ] }, { "cell_type": "markdown", "id": "59fdaee1", "metadata": {}, "source": [ "`pydantic` is a library that makes it easy to define and validate data in Python.\n", "It can also parse data from various formats and serialize it to various formats." ] }, { "cell_type": "code", "execution_count": 4, "id": "eb9f66c2", "metadata": {}, "outputs": [], "source": [ "from pydantic import BaseModel, Field\n", "from typing import List" ] }, { "cell_type": "code", "execution_count": 7, "id": "b0104cd4", "metadata": {}, "outputs": [], "source": [ "class Reaction(BaseModel):\n", " reaction_name: str = Field(..., description=\"The name of the reaction\")\n", " reactants: List[str] = Field(..., description=\"The reactants of the reaction\")\n", " catalyst: List[str] = Field(..., description=\"The catalysts of the reaction\")\n", " base: str = Field(..., description=\"The base of the reaction\")\n", " solvent: str = Field(..., description=\"The solvent of the reaction\")\n", " temperature: int = Field(..., description=\"The temperature of the reaction\")\n", " temperature_unit: str = Field(..., description=\"The unit of the temperature\")\n", " product: str = Field(..., description=\"The product of the reaction\")\n", " rxn_yield: float = Field(..., description=\"The yield of the reaction\")" ] }, { "cell_type": "markdown", "id": "4dbf3086", "metadata": {}, "source": [ "We can now use the `Reaction` class to create an instance of the `Reaction` using a dict." ] }, { "cell_type": "code", "execution_count": 9, "id": "fb1718446ad10c60", "metadata": { "collapsed": false }, "outputs": [], "source": [ "rxn_dict = {\n", " \"reaction_name\": \"Buchwald-Hartwig reaction\",\n", " \"reactants\": [\"5-Bromo-m-xylene\", \"Benzylmethylamine\"],\n", " \"catalyst\": [\"Bis(dibenzylideneacetone)palladium(0)\", \"Tri(o-tolyl)phosphine\"],\n", " \"base\": \"Sodium tert-butoxide\",\n", " \"solvent\": \"Toluene\",\n", " \"temperature\": 65,\n", " \"temperature_unit\": \"°C\",\n", " \"product\": \"N-Benzyl-N-methyl(3,5-xylyl)amine\",\n", " \"rxn_yield\": 88,\n", "}" ] }, { "cell_type": "code", "execution_count": 11, "id": "f1457921", "metadata": {}, "outputs": [], "source": [ "rxn = Reaction(**rxn_dict)" ] }, { "cell_type": "markdown", "id": "4b69de0f", "metadata": {}, "source": [ "We can also save the data as a JSON file." ] }, { "cell_type": "code", "execution_count": 15, "id": "6c8b0a90", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'{\"reaction_name\":\"Buchwald-Hartwig reaction\",\"reactants\":[\"5-Bromo-m-xylene\",\"Benzylmethylamine\"],\"catalyst\":[\"Bis(dibenzylideneacetone)palladium(0)\",\"Tri(o-tolyl)phosphine\"],\"base\":\"Sodium tert-butoxide\",\"solvent\":\"Toluene\",\"temperature\":65,\"temperature_unit\":\"°C\",\"product\":\"N-Benzyl-N-methyl(3,5-xylyl)amine\",\"rxn_yield\":88.0}'" ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" } ], "source": [ "rxn.model_dump_json()" ] }, { "cell_type": "markdown", "id": "f80607f7e41c0f75", "metadata": { "collapsed": false }, "source": [ "```{admonition} Annotating data\n", ":class: tip\n", "\n", "The creation of a test and validation set is crucial and can be time-consuming. Therefore, [using an automation tool is recommended](https://hamel.dev/notes/llm/finetuning/04_data_cleaning.html). For doing so, [TeamTat](https://www.teamtat.org) can be a convenient choice.\n", "\n", "Labeling itself can be very difficult and it can be worth to consider [guidelines used by other projects](https://eugeneyan.com/writing/labeling-guidelines/).\n", "```" ] }, { "cell_type": "markdown", "id": "c477070e", "metadata": {}, "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.19" } }, "nbformat": 4, "nbformat_minor": 5 }