Deciding how to implement a data parser

There are many ways of importing data into a Simplicité App. Let’s get en overview and settle the pro’s and con’s.

1 - Adapters

1.1 - XML Adapters

1.1.1 - Standard XML import

As we know, the mother tongue of Simplicité is XML. As a reminder, here is the simplified general XML structure:

Click here to see XML
<simplicite>
    <object>
        <name>MyObjectName</name>
        <action>update</action>
        <data>
            <myObjectField>data_A1</myObjectField>
            <myObjectField2>data_A2</myObjectField2>
        </data>
        <data>
            <myObjectField>data_B1</myObjectField>
            <myObjectField2>data_B2</myObjectField2>
        </data>
    </object>
</simplicite>

In this most basic case the import looks like this:

XML_Simplicité —> nativeXMLReader() —> DB

One good thing is that with this native reader this data travels through all the normal validations pipes and Java hooks that you configured for your object. It’s just as if you copied it on the forms and hit “save”, if the data is not valid, you’ll get an error.

1.1.2- Stock XML adapters

Adapters are called the way they are called because historically they were concieved as functions that would transform a specific kind of structured data (for example JSON) to the Simplicité-XML format:

JSON —> stockJsonAdapter() —> Simplicité-XML —> nativeXMLReader() —> DB

Simplicité comes bundled with some “stock adapters”. They are structurally identical to the native XML format:

  • JSON adapter
  • YAML adapter
  • ZIP adapter (files are bundled with the XML instead of being inlined into it as base64)
  • CSV adapter (deprecated)

1.1.3 - Custom XML Adapters

Most of the time, the standard XML/JSON structure is not the needed format, so you need to write your own adapter:

customStructuredData —> customAdapter() —> Simplicité-XML —> nativeXMLReader() —> DB

Often, these custom adpaters leverage Simplicité’s parsing mechanisms by inheriting one of the internal classes, all in the com.simplicite.util.integration package :

Click to see internal class structure
  • SimpleAdapter
    • LineBasedAdapter
      • CSVLineBasedAdapter
    • SAXParserAdapter
      • SimpleSAXParserAdapter
        • SAXImportXML
    • SimpleJSONAdapter
      • SimpleYAMLAdapter
    • SimpleXLSAdapter
      • CellBasedXLSAdapter
    • SimpleXLSXAdapter
      • CellBasedXLSXAdapter
    • SimpleSQLAdapter
      • SQLAdapter

To create a new adapter, go to Operation > Adapter > Create. Once created (and the Java code written):

  • the adapter will be automatically made available through the XML Import page => you don’t need to code any UI.
  • in the Imports Supervisor object, your adapter will generate will automatically save:
    • the input file
    • the XML generated by your adapter
    • a log file, concatening:
      • our adapter logs
      • the native XML reader logs
    • an error file, if the XML import fails

The following example transforms presents a basic CSV->XML Adapter:

Click to see example

Demo

csvadapter

Java Adapter Code:

package com.simplicite.adapters.Application;
import java.util.*;
import com.simplicite.util.integration.*;

public class SampleCsvToXmlAdapter extends CSVLineBasedAdapter {
	private static final long serialVersionUID = 1L;
	
	public String preProcess(){
		appendLog("Starting SampleCsvToXmlAdapter");
		// set CSV separator
		setSeparator(','); 

		// to generate a subsequently imported XML, call super.preProcess()
		// doing so will add a starting <simplicite> tag
		return super.preProcess();
	}
	
	@Override
	public String processValues(long lineNumber, String[] values){
		appendLog("=== Processing line #"+lineNumber+" : "+Arrays.toString(values));
		
		StringBuilder xml = new StringBuilder();
		xml.append("\t<object>\n");
		xml.append("\t\t<name>DemoSupplier</name>\n");
		xml.append("\t\t<action>upsert</action>\n");
		xml.append("\t\t<data>\n");
		xml.append("\t\t\t<demoSupCode>"+values[0]+"</demoSupCode>\n");
		xml.append("\t\t\t<demoSupName>"+values[1]+"</demoSupName>\n");
		xml.append("\t\t</data>\n");
		xml.append("\t</object>\n");
		
		// returned String gets added to a XML subsequently imported.
		return xml.toString(); 		
	}

	public void postProcess(){
		appendLog("End Process with status "+getStatus());
		// to generate a subsequently imported XML, call super.postProcess()
		// doing so will add a closing <simplicite> tag
		super.postProcess();
	}
}

Input CSV

codeSupplierA,nameSupplierA
codeSupplierB,nameSupplierB

Resulting XML

<?xml version="1.0" encoding="UTF-8"?>
<simplicite xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://www.simplicite.fr/base" xsi:schemaLocation="http://www.simplicite.fr/base https://www.simplicite.io/resources/schemas/base.xsd">
	<object>
		<name>DemoSupplier</name>
		<action>upsert</action>
		<data>
			<demoSupCode>codeSupplierA</demoSupCode>
			<demoSupName>nameSupplierA</demoSupName>
		</data>
	</object>
	<object>
		<name>DemoSupplier</name>
		<action>upsert</action>
		<data>
			<demoSupCode>codeSupplierB</demoSupCode>
			<demoSupName>nameSupplierB</demoSupName>
		</data>
	</object>
</simplicite>

See more example and explanations about the helper classes in the docs : Simplicité® documentation/01-core/adapter-code-examples

1.2 - Direct Adapters

There are cases where you don’t want to generate the intermediate XML. Think about very large imports, that would clutter the import supervisor object with huge XML files, and might pose memory problems. In these cases, we can use direct adapters, for which the usage is the same, but the way of processing the data is different:

customStructuredData —> customAdapter() —> DB

These adapter can leverage the same helper classes refered before: Simplicité® documentation/01-core/adapter-code-examples

A concise example is available here: CSV Adapter Example

1.3 - Stock CSV Adapter

A special case is the new Stock CSV Importer, that will offer an admin-friendly interface to import CSV files and apply some mappings and transformations to the data. It’s still limited to one object per CSV but has two modes : direct (CSV=>DB) or indirect (CSV=>XML=>DB)

2 - Custom Import Objects

In cases where…

  • you don’t need/don’t want to clutter the Import Supervisor,
  • OR need your non technical users to import the data (no access to “Operations”)
  • OR need custom logging / error messages available to business
  • OR need a link between the import and the business object

…you might need to build a custom import object.

Below is a screenshot of such a case, where the imported objects must be systematically associated to a source file and follow a manual validation state model (“Imported” > “Validated”), done either by record or in bulk. The import process is coded in the “import object” and is totally independent from the adapter mechanisms; let’s note that it could leverage an adapter through code if we want to also use the import supervisor & parallelisation features.

One last word…

Wether you use an adapter or a custom import object, it’s always Java runing under the hood that you’ll have to code, please make sur you understand how parsers work and why you should one against another. These choices can have big memory and performance implications.