Import Interrupted

Bonjour,

Voici la version et la révision de l’instance

[Platform]
Status=OK
Version=4.0.P24
BuiltOn=2020-03-05 23:18 (revision 40c7997ef4ecb7b5cf073d0427e387a478b0eb96)
Encoding=UTF-8
EndpointIP=172.17.0.7
EndpointURL=http://b7fd5ea8e9b2:8080
TimeZone=Europe/Paris
SystemDate=2020-03-26 11:02:42

J’essaie d’importer des données (fichier txt de 26000 lignes) en utilisant un adapter codé en java. Le problème est que cet import s’interrompt au bout d’une centaine de lignes parfois, parfois au bout de 1000 lignes (parfois un peu plus). Il reste cependant in progress (mais en suivant les données importées je vois qu’il est stoppé). je dois donc le mettre à KO pour réimporter à nouveau.
Je n’arrive pas à trouver une raison pour cet incident. Auriez vous des pistes? des idées?

Merci de votre aide

Sans votre code on ne peut pas vous aider

code adapter


import java.util.List;
import java.util.ArrayList;
import java.util.Date;
import java.text.SimpleDateFormat;
import java.util.HashMap;
import java.util.Map;

import org.json.JSONObject;

import com.simplicite.objects.RenaultSites.SitesAccessPoint;
import com.simplicite.objects.RenaultSites.SitesAccessPointSitesAddress;
import com.simplicite.objects.RenaultSites.SitesAccessPointIdentifierValue;
import com.simplicite.objects.RenaultSites.SitesActivity;
import com.simplicite.objects.RenaultSites.SitesAssignment;
import com.simplicite.objects.RenaultSites.SitesCategory;
import com.simplicite.objects.RenaultSites.SitesGeographicalAddress;
import com.simplicite.objects.RenaultSites.SitesGeolocationCoordinates;
import com.simplicite.objects.RenaultSites.SitesPostalAddress;
import com.simplicite.objects.RenaultSites.SitesRealEstateSite;
import com.simplicite.objects.RenaultSites.SitesRealEstateAssetIdentifierValue;
import com.simplicite.objects.RenaultSites.SitesSiteGroupingAssignment;
import com.simplicite.util.ObjectField;
import com.simplicite.util.Tool;
import com.simplicite.util.tools.BusinessObjectTool;

public class SitesRaftAD extends com.simplicite.commons.RenaultSites.SitesCommonAdapter {
	private static final long serialVersionUID = 1L;

	private boolean debug = false;
	private String SUPPLIER_SITE_CATEGORY = "B06";
	private String SUPPLIER_SITE_ACTIVITY = "A10";
	private String SITE_PHYSICAL_STATE = "Built";

	protected SitesAccessPoint acp;
	private SitesActivity act;
	private SitesGeographicalAddress aga;
	private SitesGeolocationCoordinates agc;
	private SitesPostalAddress aps;
	private SitesAssignment asg;
	private SitesCategory sca;
	private SitesRealEstateSite res;
	private SitesRealEstateAssetIdentifierValue idv;
	private SitesSiteGroupingAssignment sga;
	private ArrayList<String> activeSites = new ArrayList<>();
	// Archived sites and its access points
	private HashMap<String, ArrayList<String>> archivedSitesAccessPoints = new HashMap<>();

	protected transient BusinessObjectTool acpTool;
	private transient BusinessObjectTool actTool;
	private transient BusinessObjectTool acpAdrTool;
	private transient BusinessObjectTool agcTool;
	private transient BusinessObjectTool agaTool;
	private transient BusinessObjectTool apsTool;
	private transient BusinessObjectTool asgTool;
	private transient BusinessObjectTool scaTool;
	private transient BusinessObjectTool resTool;
	private transient BusinessObjectTool idvTool;
	private transient BusinessObjectTool sgaTool;

	@Override
	public String preProcess() {
		acp = (SitesAccessPoint) getGrant().getTmpObject("SitesAccessPoint");
		acpTool = new BusinessObjectTool(acp);
		act = (SitesActivity) getGrant().getTmpObject("SitesActivity");
		actTool = new BusinessObjectTool(act);
		agc = (SitesGeolocationCoordinates) getGrant().getTmpObject("SitesGeolocationCoordinates");
		agcTool = new BusinessObjectTool(agc);
		aga = (SitesGeographicalAddress) getGrant().getTmpObject("SitesGeographicalAddress");
		agaTool = new BusinessObjectTool(aga);
		aps = (SitesPostalAddress) getGrant().getTmpObject("SitesPostalAddress");
		apsTool = new BusinessObjectTool(aps);
		asg = (SitesAssignment) getGrant().getTmpObject("SitesAssignment");
		asgTool = new BusinessObjectTool(asg);		
		sca = (SitesCategory) getGrant().getTmpObject("SitesCategory");
		scaTool = new BusinessObjectTool(sca);
		res = (SitesRealEstateSite) getGrant().getTmpObject("SitesRealEstateSite");
		resTool = new BusinessObjectTool(res);
		idv = (SitesRealEstateAssetIdentifierValue) getGrant().getTmpObject("SitesRealEstateAssetIdentifierValue");
		idvTool = new BusinessObjectTool(idv);
		sga = (SitesSiteGroupingAssignment) getGrant().getTmpObject("SitesSiteGroupingAssignment");
		sgaTool = new BusinessObjectTool(sga);
		asg = (SitesAssignment) getGrant().getTmpObject("SitesAssignment");
		asgTool = new BusinessObjectTool(asg);		

		return super.preProcess();
	}

	/** Mapping:
Datheur                1   26  ignorer 
Type of data           27  8   ignorer
Type of update         35  1   ‘ ’ = INIT; ‘C’ = Creation; ‘M’ = Modification; ‘S’ = Archive du site
Account                36  6   Business identifier Access Point "RAFT_ACCOUNT"  
Sub-Code               42  2   Supplier site code   Business identifier ACP "RAFT_CODE" composé de la valeur de la ligne précédente (account) concaténé à la valeur de cette ligne (sub-code) 
Blank                  44  5   ignorer
Account                49  6   ignorer
Address Sub-Code       55  2   ignorer
Address 1              57  35  Postal Address ligne 1 
Address 2              92  35  Postal Address ligne 2 
Address 3              127 35  Postal Address ligne 3 
Address 4              162 35  Postal Address ligne 4 
Address 5              197 35  Postal Address ligne 5   
Postal code            232 9   CIT postal code + AGA (adresse georgraphique) 
Distributor Office     241 26  CIT name + AGA 
Province Code          267 4   AGA territorial division 
Country Code           271 2   Cty code (lié à Cit) 
Transmitter Code       273 6   Business identifier Access Point TRANS_CODE
Receptor Code          279 6   Business identifier Access Point RECEPT_CODE
Qualifier              285 50  10 occurs of 5 characters => Access Point Type 
Supplier Type          335 3   Business identifier Access Point 
Renault Country Code   338 3   ignorer 
SAGA suppress date     341 10  dd.mmd.yyyy  End date ACP + status archivé de l'ACP 
Reference Site         351 8   business identifier ACP  GPI_CODE
Country Designation    359 20  CTY 
Short social reason    379 15  Business identifier SOCIAL_RAISON
SNC Code               394 6   Business identifier SNC_CODE
Other transmitter Code 400 5   Business identifier NISSAN_CODE
CKD Indicator          405 1   Business identifier CKD_INDICATOR
Incoterm               406 3   Bisiness identifier CODE_INCOTERM
?                      409 6   ignorer
Incremental number     415 5   ignorer
      
Site NAME *Mot directeur + nom de ville
*Mot directeur = premiers caractères de Short social reason jusqu'à la première espace ou l'intégralité des 10 premiers caractères le cas échéant
	*/

	@Override
	public String processValues(long lineNumber, String[] values) {
		long beginTimestamp = 0;
		if (debug) {
			beginTimestamp = System.currentTimeMillis();
		}
		try {
			// Length check
			if (values[0].length() != 419) {
				//throw new Exception("Line " + lineNumber + " is malformed, thus ignored. values[0].length() = " + values[0].length());
			}

			if (debug) {
				appendLog("Line " + lineNumber);
				for (String value : values) {
					appendLog("\t" + value);
				}
			}

			boolean updating = false;
			String typeOfUpdate = values[0].substring(34, 35); // ‘ ’ = INIT; ‘C’ = Creation; ‘M’ = Modification; ‘S’ = Archive du site
			String raftAccount = values[0].substring(35, 41).trim(); // Business identifier Access Point "RAFT_ACCOUNT" 
			String subCode = values[0].substring(41, 43).trim(); // Business identifier ACP "RAFT_CODE" composé de la valeur de la ligne précédente (account) concaténé à la valeur de cette ligne (sub-code) 
			String address1 = values[0].substring(56, 91).trim(); // Postal Address ligne 1 
			String address2 = values[0].substring(91, 126).trim(); // Postal Address ligne 2 
			String address3 = values[0].substring(126, 161).trim(); // Postal Address ligne 3 
			String address4 = values[0].substring(161, 196).trim(); // Postal Address ligne 4 
			String address5 = values[0].substring(196, 231).trim(); // Postal Address ligne 5 
			String postalCode = values[0].substring(231, 240).trim(); // CIT postal code + AGA (adresse georgraphique) 
			String distributorOffice = values[0].substring(240, 266).trim(); // CIT name + AGA 
			String provinceCode = values[0].substring(266, 270).trim(); // AGA territorial division 
			String countryCode = values[0].substring(270, 272).trim(); // Cty code (lié à Cit) 
			String transmitterCode = values[0].substring(272, 278).trim(); // Business identifier Access Point TRANS_CODE
			String receptorCode = values[0].substring(278, 284).trim(); // Business identifier Access Point RECEPT_CODE
			String qualifier = values[0].substring(284, 334); // 10 occurs of 5 characters => Access Point Type 
			String supplierType = values[0].substring(334, 337).trim(); // Business identifier Access Point 
			String raftCountryCode = values[0].substring(337, 340).trim(); // Business identifier Access Point 
			String sagaSuppDate = values[0].substring(340, 350).trim(); // dd.mmd.yyyy End date ACP + status archivé de l'ACP 
			String referenceSite = values[0].substring(350, 358).trim(); // business identifier ACP GPI_CODE
			String raftCountryName = values[0].substring(358, 378).trim(); // CTY 
			String socialReason = values[0].substring(378, 393).trim(); // Business identifier SOCIAL_RAISON
			String sncCode = values[0].substring(393, 399).trim(); // Business identifier SNC_CODE
			String otherTransmitterCode = values[0].substring(399, 404).trim(); // Business identifier NISSAN_CODE
			String ckdIndicator = values[0].substring(404, 405).trim(); // Business identifier CKD_INDICATOR
			String incoterm = values[0].substring(405, 408).trim(); // Bisiness identifier CODE_INCOTERM
			String raftCode = raftAccount + subCode;
			//String shortSocialReason = socialReason.substring(0, 10).trim();
			String shortSocialReason = values[0].substring(419).trim();
			String siteName = shortSocialReason + " " + distributorOffice;

			/**
			 *
			 * Retrieving site
			 *
			 */

			String resId = null;
			switch(typeOfUpdate) {
				case " ": // creation
				case "C": // creation
					resId = res.getRowIdFromAlternativeIdentifier(siteName.toUpperCase(), "RAFT_SITENAME", false);
					if (!Tool.isEmpty(resId)) {
						updating = true;
						res.select(resId);
					}
					else {
						res.resetValues(true);
					}
					break;
				case "M": // update
				case "S": // archival
					updating = true;
					resId = res.getRowIdFromAlternativeIdentifier(siteName.toUpperCase(), "RAFT_SITENAME", false);
					if (!Tool.isEmpty(resId)) {
						res.select(resId);
					}
					else {
						throw new Exception("typeOfUpdate is set to udpate/archive so there should have been an existing site but no resId found on line: " + lineNumber);
					}
					break;
			}
			res.setFieldValue("sitesReaName", siteName);
			res.setFieldValue("sitesReaPhysicalState", SITE_PHYSICAL_STATE);
			res.setFieldValue("sitesGenStatus", "ACTIVE");
			res.setFieldValue("sitesReaEndDate", ""); // in case the site has been referenced as archived by a previous line
			resTool.validateAndSave();
			resId = res.getRowId();
			if (sagaSuppDate.isEmpty()) {
				activeSites.add(resId);
			}

			/**
			 *
			 * Site Category assignment
			 *
			 */
			String scaId = getCategoryId(SUPPLIER_SITE_CATEGORY);
			if (scaId != null) {
				try {
					List<String[]> rows = asgTool.search(new JSONObject().put("sitesAsgReaId", resId).put("sitesAsgScaId", scaId));
					if (rows.size() == 0) { // Create if not exists (no need to update)
						asg.resetValues(true);
						asg.setFieldValue("sitesAsgReaId", resId);
						asg.setFieldValue("sitesAsgScaId", scaId);
						asgTool.validateAndSave();
					}
				} catch (Exception e) {
					warning("processValues", "Unable to save category for site ID " + resId + " and category ID " + scaId, e);
				}
			} else {
				throw new Exception("Category is not defined in the target environment, on line: " + lineNumber);
			}

			/**
			 *
			 * Site Activity assignment
			 *
			 */
			String actId = getActivityId(SUPPLIER_SITE_ACTIVITY);
			if (actId != null) {
				try {
					List<String[]> rows = asgTool.search(new JSONObject().put("sitesAsgReaId", resId).put("sitesAsgActId", actId));
					if (rows.size() == 0) { // Create if not exists (no need to update)
						asg.resetValues(true);
						asg.setFieldValue("sitesAsgReaId", resId);
						asg.setFieldValue("sitesAsgActId", actId);
						asgTool.validateAndSave();
					}
				} catch (Exception e) {
					warning("processValues", "Unable to save category for site ID " + resId + " and activity ID " + actId, e);
				}
			} else {
				throw new Exception("Activity is not defined in the target environment, on line: " + lineNumber);
			}


			/**
			 *
			 * Addresses
			 *
			 */
			String agaId = null;
			String apsId = null;
			String agaSummary = null;
			String apsSummary = null;
			String ctyId = getCountryId(countryCode);
			if (ctyId == null) {
				throw new Exception("No country found for code: " + countryCode);
			}
			String citId = createOrUpdateCity(ctyId, postalCode, distributorOffice, countryCode, resId);

			// Create postal address
			try {
				aps.resetValues(true);
				if (!address1.isEmpty()) {
					aps.setFieldValue("sitesApsLine1", address1);
				}
				if (!address2.isEmpty()) {
					aps.setFieldValue("sitesApsLine2", address2);
				}
				if (!address3.isEmpty()) {
					aps.setFieldValue("sitesApsLine3", address3);
				}
				if (!address4.isEmpty()) {
					aps.setFieldValue("sitesApsLine4", address4);
				}
				if (!address5.isEmpty()) {
					aps.setFieldValue("sitesApsLine5", address5);
				}
				if (!postalCode.isEmpty()) {
					aps.setFieldValue("sitesApsLine6", postalCode);
				}
				if (!distributorOffice.isEmpty()) {
					aps.setFieldValue("sitesApsLine7", distributorOffice);
				}
				apsTool.validateAndSave();
				apsId = aps.getRowId();
				apsSummary = aps.getFieldValue("sitesAdrSummary");
			} catch (Exception e) {
				warning("processValues", "Unable to save postal address for raft code " + raftCode, e);
			}
			if (debug) {
				appendLog("apsId = " + apsId);
			}

			// Find or create geographical address
			try {
				aga.resetValues(true);
				if (citId != null) {
					aga.populateForeignKey("sitesAgaCitId", citId, false, true); // Set ID + retreive referenced data (because they are used by preSave hook of geographical address)
				}
				if (!distributorOffice.isEmpty()) {
					aga.setFieldValue("sitesAgaLocality", distributorOffice);
				}
				if (!postalCode.isEmpty()) {
					aga.setFieldValue("sitesAgaPostalCode", postalCode);
				}
				if (!provinceCode.isEmpty()) {
					aga.setFieldValue("sitesAgaTerritorialDivision", provinceCode);
				}
				agaTool.validateAndSave();
				agaId = aga.getRowId();
				agaSummary = aga.getFieldValue("sitesAdrSummary");
			} catch (Exception e) {
				warning("processValues", "Unable to create geographical address", e);
			}
			if (debug) {
				appendLog("agaId = " + agaId);
			}

			/**
			 *
			 * Create access points
			 *
			 */

			ArrayList<String> accessPoints = new ArrayList<>();
			int nbAcpCreated = 0;

			for (int i = 0; i < 10; i++) {
				String acpType;
				switch (qualifier.substring(5 * i, 5 * i + 5).trim()) {
					case "FAB":
						acpType = "P05";
						break;
					case "FACTU":
						acpType = "P06";
						break;
					case "ENLEV":
						acpType = "P07";
						break;
					case "SICLI":
						acpType = "P08";
						break;
					case "EXPED":
						acpType = "P09";
						break;
					case "COMAN":
						acpType = "P11";
						break;
					case "CDG":
						acpType = "P13";
						break;
					case "FABT":
						acpType = "P14";
						break;
					default:
						acpType = null;
				}
				if (acpType != null) {
					String acpId = null;
					try {
						acpId = createAccessPoint(resId, acpType, "SitesRaftAD");
					} catch (Exception e) {
						throw new Exception("Unable to save access point for site ID " + resId);
					}
					nbAcpCreated++;
					if (apsId != null) {
						createAccessPointAddress(acpId, apsId, "POSTAL", apsSummary, "SitesRaftAD");
					}
					if (agaId != null) {
						createAccessPointAddress(acpId, agaId, "GEOGRAPHICAL", agaSummary, "SitesRaftAD");
					}
					accessPoints.add(acpId);
				}
			}
			// If no qualifier: creating an access point "other" (P04) with the line's address
			if (accessPoints.size() == 0) {
				String acpId = null;
				try {
					acpId = createAccessPoint(resId, "P04", "SitesRaftAD");
				} catch (Exception e) {
					throw new Exception("Unable to save access point for site ID " + resId);
				}
				if (apsId != null) {
					createAccessPointAddress(acpId, apsId, "POSTAL", apsSummary, "SitesRaftAD");
				}
				if (agaId != null) {
					createAccessPointAddress(acpId, agaId, "GEOGRAPHICAL", agaSummary, "SitesRaftAD");
				}
				accessPoints.add(acpId);
			}

			/** 
			 *
			 * Archiving site and access points
			 *
			 */

			// A site is archived if there's a sagaSuppDate for one line referencing the site 
			// AND none of the lines references the site as "ACTIVE"
			// = no sagaSuppDate for this line 
			// = this line not contained in the activeSites list)
			if (!sagaSuppDate.isEmpty() && !activeSites.contains(resId)) {
				String DD = sagaSuppDate.substring(0, 2);
				String MM = sagaSuppDate.substring(3, 5);
				String YYYY = sagaSuppDate.substring(6);
				String endDate = YYYY + "-" + MM + "-" + DD;
				res.setFieldValue("sitesReaEndDate", endDate);
				res.setFieldValue("sitesGenStatus", "ARCHIVED");
				for (String acpId: accessPoints) {
					acp.select(acpId);
					acp.setFieldValue("sitesGenStatus", "ARCHIVED");
					acp.setFieldValue("sitesAcpEndDate", endDate);
					ArrayList<String> archivedAccessPoints = archivedSitesAccessPoints.get(resId);
					if (archivedAccessPoints == null) {
						archivedAccessPoints = new ArrayList<>();
					}
					archivedAccessPoints.add(acpId);
					acpTool.validateAndSave();
				}
				resTool.validateAndSave();
			}
			else {
				ArrayList<String> archivedAccessPoints = archivedSitesAccessPoints.get(resId);
				if (archivedAccessPoints != null) {
					for (String acpId: archivedAccessPoints) {
						acp.select(acpId);
						acp.setFieldValue("sitesGenStatus", "ACTIVE");
						acpTool.validateAndSave();
					}					
				}
			}

			/**
			 *
			 * Business identifiers
			 *
			 */

			createOrUpdateRealEstateAssetIdentifier("RAFT_ACCOUNT", resId, raftAccount, "RAFT");
			createOrUpdateRealEstateAssetIdentifier("RAFT_SITENAME", resId, siteName, "RAFT");

			for (String acpId: accessPoints) {
				if (!raftAccount.isEmpty()) {
					createOrUpdateAccessPointIdentifier("RAFT_ACCOUNT", acpId, raftAccount, "RAFT");
				}
				if (!raftCode.isEmpty()) {
					createOrUpdateAccessPointIdentifier("RAFT_CODE", acpId, raftCode, "RAFT");
				}
				if (!transmitterCode.isEmpty()) {
					createOrUpdateAccessPointIdentifier("TRANS_CODE", acpId, transmitterCode, "RAFT");
				}
				if (!receptorCode.isEmpty()) {
					createOrUpdateAccessPointIdentifier("RECEPT_CODE", acpId, receptorCode, "RAFT");
				}
				if (!supplierType.isEmpty()) {
					createOrUpdateAccessPointIdentifier("SUPPLIER_TYPE", acpId, supplierType, "RAFT");
				}
				if (!referenceSite.isEmpty()) {
					createOrUpdateAccessPointIdentifier("GPI_CODE", acpId, referenceSite, "RAFT");
				}
				if (!socialReason.isEmpty()) {
					createOrUpdateAccessPointIdentifier("SOCIAL_RAISON", acpId, socialReason, "RAFT");
				}
				if (!sncCode.isEmpty()) {
					createOrUpdateAccessPointIdentifier("SNC_CODE", acpId, sncCode, "RAFT");
				}
				if (!otherTransmitterCode.isEmpty()) {
					createOrUpdateAccessPointIdentifier("NISSAN_CODE", acpId, otherTransmitterCode, "RAFT");
				}
				if (!ckdIndicator.isEmpty()) {
					createOrUpdateAccessPointIdentifier("CKD_INDICATOR", acpId, ckdIndicator, "RAFT");
				}
				if (!incoterm.isEmpty()) {
					createOrUpdateAccessPointIdentifier("CODE_INCOTERM", acpId, incoterm, "RAFT");
				}
			}

		} catch (Exception e) {
			appendLog(e.getMessage());
			if (debug) {
				appendLog("Elapsed time (ms): " + (System.currentTimeMillis() - beginTimestamp));
			}
			appendError(values); // Mark current line as rejected
		}
		nbLines++;
		if (debug) {
			appendLog("Elapsed time (ms): " + (System.currentTimeMillis() - beginTimestamp));
		}
		return null;
	}

	@Override
	public void postProcess() {
		super.postProcess();
	}
}

code commun

package com.simplicite.commons.RenaultSites;

import java.util.HashMap;
import java.util.List;
import java.util.ArrayList;

import org.json.JSONObject;

import com.simplicite.objects.RenaultSites.SitesAccessPoint;
import com.simplicite.objects.RenaultSites.SitesAccessPointIdentifierValue;
import com.simplicite.objects.RenaultSites.SitesAccessPointSitesAddress;
import com.simplicite.objects.RenaultSites.SitesActivity;
import com.simplicite.objects.RenaultSites.SitesCategory;
import com.simplicite.objects.RenaultSites.SitesCity;
import com.simplicite.objects.RenaultSites.SitesCountry;
import com.simplicite.objects.RenaultSites.SitesEmailAddress;
import com.simplicite.objects.RenaultSites.SitesGeographicalAddress;
import com.simplicite.objects.RenaultSites.SitesIdentifierType;
import com.simplicite.objects.RenaultSites.SitesLineOfBusiness;
import com.simplicite.objects.RenaultSites.SitesLegalEntity;
import com.simplicite.objects.RenaultSites.SitesRealEstateAssetIdentifierValue;
import com.simplicite.objects.RenaultSites.SitesRealEstateSite;
import com.simplicite.objects.RenaultSites.SitesSiteGroupingElement;
import com.simplicite.objects.RenaultSites.SitesSourceSystem;
import com.simplicite.objects.RenaultSites.SitesTelephoneAddress;
import com.simplicite.objects.RenaultSites.SitesWebAddress;
import com.simplicite.util.AppLog;
import com.simplicite.util.tools.BusinessObjectTool;
import com.simplicite.util.ObjectField;
import com.simplicite.util.Tool;

/**
 * Common adapter
 */
public abstract class SitesCommonAdapter extends com.simplicite.util.integration.CSVLineBasedAdapter {
	private static final long serialVersionUID = 1L;

	protected boolean debug = false;

	protected SitesAccessPoint acp;
	protected SitesAccessPointIdentifierValue api;
	protected SitesAccessPointSitesAddress acpAdr;
	protected SitesActivity act;
	protected SitesCity cit;
	protected SitesCountry cty;
	protected SitesEmailAddress aem;
	protected SitesGeographicalAddress aga;
	protected SitesIdentifierType idt;
	protected SitesLegalEntity lge;
	protected SitesLineOfBusiness lob;
	protected SitesRealEstateAssetIdentifierValue idv;
	protected SitesRealEstateSite res;
	protected SitesCategory sca;
	protected SitesSiteGroupingElement sgr;
	protected SitesSourceSystem sys;
	protected SitesTelephoneAddress ate;
	protected SitesWebAddress awe;
	protected transient BusinessObjectTool acpAdrTool;
	protected transient BusinessObjectTool acpTool;
	protected transient BusinessObjectTool actTool;
	protected transient BusinessObjectTool aemTool;
	protected transient BusinessObjectTool agaTool;
	protected transient BusinessObjectTool apiTool;
	protected transient BusinessObjectTool ateTool;
	protected transient BusinessObjectTool aweTool;
	protected transient BusinessObjectTool citTool;
	protected transient BusinessObjectTool ctyTool;
	protected transient BusinessObjectTool idtTool;
	protected transient BusinessObjectTool idvTool;
	protected transient BusinessObjectTool lgeTool;
	protected transient BusinessObjectTool lobTool;
	protected transient BusinessObjectTool resTool;	
	protected transient BusinessObjectTool scaTool;
	protected transient BusinessObjectTool sgrTool;
	protected transient BusinessObjectTool sysTool;

	protected int nbLines = 0;
	protected int nbWarnings = 0;
	protected int nbErrors = 0;

	/** Log info to logger and to adapter log */
	protected void info(String meth, String msg) {
		AppLog.info(getClass(), meth, msg, getGrant());
		if (msg!=null) appendLog("Info: " + msg);
	}

	/** Log warning to logger and to adapter log */
	protected void warning(String meth, String msg, Exception e) {
		if (msg==null && e!=null) msg = e.getMessage();
		AppLog.warning(getClass(), meth, msg, e, getGrant());
		if (msg!=null) appendLog("Warning: " + msg);
		nbWarnings++;
	}

	/** Log error to logger and to adapter log */
	protected void error(String meth, String msg, Exception e) {
		if (msg==null && e!=null) msg = e.getMessage();
		AppLog.error(getClass(), meth, msg, e, getGrant());
		if (msg!=null) appendLog("Error: " + msg);
		nbErrors++;
	}

	/** Clean string */
	protected String cleanString(String s, boolean extended, boolean uppercase) {
		s = s==null ? "" : s.trim();
		if (extended) s = s.replace('-', ' ');
		return (uppercase ? s.toUpperCase() : s).replaceAll("\\s+", " "); 
	}

	private HashMap<String, String> sourceSystems = new HashMap<>();
	protected String getSourceSystemId(String name) {
		String id = sourceSystems.get(name);
		if (id == null) {
			try {
				id = sysTool.search(new JSONObject().put("sitesSysName", name)).get(0)[0];
				sourceSystems.put(name, id);
				if (debug) appendLog("Found source systems for " + name + ": " + id);
			} catch (Exception e) {
				if (debug) appendLog("Unable to find source systems for " + name + " (" + e.getMessage() + ")");
			}
		}
		return id;
	}

	private HashMap<String, String> identifierTypes = new HashMap<>();
	protected String getIdentifierTypeId(String name) {
		String id = identifierTypes.get(name);
		if (id == null) {
			try {
				id = idtTool.search(new JSONObject().put("sitesIdtName", name)).get(0)[0];
				identifierTypes.put(name, id);
				if (debug) appendLog("Found identifier type for " + name + ": " + id);
			} catch (Exception e) {
				if (debug) appendLog("Unable to find identifier type for " + name + " (" + e.getMessage() + ")");
			}
		}
		return id;
	}

	private HashMap<String, String> countries = new HashMap<>();
	protected String getCountryId(String code) {
		String id = countries.get(code);
		if (id == null) {
			try {
				id = ctyTool.search(new JSONObject().put("sitesCtyCode2", code)).get(0)[0];
				countries.put(code, id);
				if (debug) appendLog("Found country for " + code + ": " + id);
			} catch (Exception e) {
				if (debug) appendLog("Unable to find country for " + code + " (" + e.getMessage() + ")");
			}
		}
		return id;
	}

	private HashMap<String, String> cities = new HashMap<>();
	protected String getCityId(String countryCode, String cityName) {
		String key = countryCode + ":" + cityName;
		String id = cities.get(key);
		if (id == null) {
			try {
				id = citTool.search(new JSONObject().put("sitesCitCtyId.sitesCtyCode2", countryCode).put("sitesCitName", cityName)).get(0)[0];
				cities.put(key, id);
				if (debug) appendLog("Found city for " + key + ": " + id);
			} catch (Exception e) {
				if (debug) appendLog("Unable to find city for " + key + " (" + e.getMessage() + ")");
			}
		}
		return id;
	}

	private HashMap<String, String> geographicalAddresses = new HashMap<>();
	protected String getGeographicAddressId(String citId, String streetName) {
		String key = citId + ":" + streetName;
		String id = geographicalAddresses.get(key);
		if (id == null) {
			try {
				id = agaTool.search(new JSONObject().put("sitesAgaCitId", citId).put("sitesAgaStreetName", streetName)).get(0)[0];
				geographicalAddresses.put(key, id);
				if (debug) appendLog("Found geographic address for " + key + ": " + id);
			} catch (Exception e) {
				if (debug) appendLog("Unable to find geographic address for " + key + " (" + e.getMessage() + ")");
			}
		}
		return id;
	}

	private HashMap<String, String> groupingElements = new HashMap<>();
	protected String getGroupingElementId(String name) {
		String id = groupingElements.get(name);
		if (id == null) {
			try {
				id = sgrTool.search(new JSONObject().put("sitesSgrName", name)).get(0)[0];
				groupingElements.put(name, id);
				if (debug) appendLog("Found grouping element for " + name + ": " + id);
			} catch (Exception e) {
				if (debug) appendLog("Unable to find grouping element for " + name + " (" + e.getMessage() + ")");
			}
		}
		return id;
	}

	private HashMap<String, String> lineOfBusinesses = new HashMap<>();
	protected String getLineOfBusinessId(String type) {
		String id = lineOfBusinesses.get(type);
		if (id == null) {
			try {
				id = lobTool.search(new JSONObject().put("sitesLobType", "%").put("sitesLobName", type)).get(0)[0];
				lineOfBusinesses.put(type, id);
				if (debug) appendLog("Found line of business for " + type + ": " + id);
			} catch (Exception e) {
				if (debug) appendLog("Unable to find line of business for " + type + " (" + e.getMessage() + ")");
			}
		}
		return id;
	}

	private HashMap<String, String> activities = new HashMap<>();
	protected String getActivityId(String name) {
		String id = activities.get(name);
		if (id == null) {
			try {
				id = actTool.search(new JSONObject().put("sitesActCode", "%").put("sitesActName", name)).get(0)[0];
				activities.put(name, id);
				if (debug) appendLog("Found activity for " + name + ": " + id);
			} catch (Exception e) {
				if (debug) appendLog("Unable to find activity for " + name + " (" + e.getMessage() + ")");
			}
		}
		return id;
	}

	private HashMap<String, String> categories = new HashMap<>();
	protected String getCategoryId(String code) {
		String id = categories.get(code);
		if (id == null) {
			try {
				id = scaTool.search(new JSONObject().put("sitesScaCode", code)).get(0)[0];
				categories.put(code, id);
				if (debug) appendLog("Found category for " + code + ": " + id);
			} catch (Exception e) {
				if (debug) appendLog("Unable to find category for " + code + " (" + e.getMessage() + ")");
			}
		}
		return id;
	}

	// TODO: add map
	protected String getLegalEntityId(String corporateName) throws Exception {
		String lgeId = null;
		try {
			lgeId = lgeTool.search(new JSONObject().put("sitesLgeCorporateName", corporateName)).get(0)[0];
			if (debug) {
				appendLog("Found legal entity for corporate name " + corporateName + ": " + lgeId);
			}
		} catch (Exception e) {
			if (debug) appendLog("Unable to find legal entity for corportate name " + corporateName + " (" + e.getMessage() + ")");
		}
		return lgeId;
	}

	protected String createLegalEntity(String corporateName) throws Exception {
		try {
			List<String[]> rows = lgeTool.search(new JSONObject().put("sitesLgeCorporateName", corporateName));
			if (rows.size() == 0) { // Create if not exists (no need to update)
				lge.resetValues(true);
				lge.setFieldValue("sitesLgeCorporateName", corporateName);
				lgeTool.validateAndSave();
			}
			return lge.getRowId();
		} catch (Exception e) {
			throw new Exception("Unable to save legal entity for corporate name " + corporateName);
		}		
	}

	@Override
	public String preProcess() {
		aga = (SitesGeographicalAddress)getGrant().getTmpObject("SitesGeographicalAddress");
		agaTool = new BusinessObjectTool(aga);

		acp = (SitesAccessPoint) getGrant().getTmpObject("SitesAccessPoint");
		acpTool = new BusinessObjectTool(acp);

		acpAdr = (SitesAccessPointSitesAddress)getGrant().getTmpObject("SitesAccessPointSitesAddress");
		acpAdrTool = new BusinessObjectTool(acpAdr);		

		act = (SitesActivity)getGrant().getTmpObject("SitesActivity");
		actTool = new BusinessObjectTool(act);

		aem = (SitesEmailAddress)getGrant().getTmpObject("SitesEmailAddress");
		aemTool = new BusinessObjectTool(aem);

		api = (SitesAccessPointIdentifierValue)getGrant().getTmpObject("SitesAccessPointIdentifierValue");
		apiTool = new BusinessObjectTool(api);
		
		ate = (SitesTelephoneAddress)getGrant().getTmpObject("SitesTelephoneAddress");
		ateTool = new BusinessObjectTool(ate);

		awe = (SitesWebAddress)getGrant().getTmpObject("SitesWebAddress");
		aweTool = new BusinessObjectTool(awe);

		cit = (SitesCity)getGrant().getTmpObject("SitesCity");
		citTool = new BusinessObjectTool(cit);

		cty = (SitesCountry)getGrant().getTmpObject("SitesCountry");
		ctyTool = new BusinessObjectTool(cty);

		idt = (SitesIdentifierType)getGrant().getTmpObject("SitesIdentifierType");
		idtTool = new BusinessObjectTool(idt);

		idv = (SitesRealEstateAssetIdentifierValue)getGrant().getTmpObject("SitesRealEstateAssetIdentifierValue");
		idvTool = new BusinessObjectTool(idv);

		lge = (SitesLegalEntity)getGrant().getTmpObject("SitesLegalEntity");
		lgeTool = new BusinessObjectTool(lge);

		lob = (SitesLineOfBusiness)getGrant().getTmpObject("SitesLineOfBusiness");
		lobTool = new BusinessObjectTool(lob);

		res = (SitesRealEstateSite) getGrant().getTmpObject("SitesRealEstateSite");
		resTool = new BusinessObjectTool(res);

		sca = (SitesCategory)getGrant().getTmpObject("SitesCategory");
		scaTool = new BusinessObjectTool(sca);

		sgr = (SitesSiteGroupingElement)getGrant().getTmpObject("SitesSiteGroupingElement");
		sgrTool = new BusinessObjectTool(sgr);
		
		sys = (SitesSourceSystem)getGrant().getTmpObject("SitesSourceSystem");
		sysTool = new BusinessObjectTool(sys);

		debug = getBooleanParameter("debug", debug); // use debug=true to enable debug

		return super.preProcess();
	}

	protected void createOrUpdateRealEstateAssetIdentifier(String identifierType, String resId, String value, String sourceSystem) throws Exception {
		String idtId = getIdentifierTypeId(identifierType);
		if (Tool.isEmpty(idtId)) {
			throw new Exception("Configuration issue: " + identifierType + " is not set up");
		}

		try {
			boolean create = false;
			List<String[]> rows = idvTool.search(new JSONObject().put("sitesIdvReaId", resId).put("sitesIdvIdtId", idtId));
			// Exists => select for update
			if (rows.size() == 1) {
				if (debug) {
					appendLog("Select for update site identifier record for found row ID = " + rows.get(0)[0]);
				}
				idv.setValues(rows.get(0));
			}
			// Doesn't exists => reset for create
			else {
				if (debug) appendLog("Reset site identifier record for creation");
				idv.resetValues(true);
				create = true;
			}
			idv.setFieldValue("sitesIdvReaId", resId);
			idv.setFieldValue("sitesIdvIdtId", idtId);
			idv.setFieldValue("sitesIdvSysId", getSourceSystemId(sourceSystem));
			idv.setFieldValue("sitesIdvValue", value);
			if (debug) {
				appendLog(new JSONObject(idv.toJSON()).toString(2));
			}
			idvTool.validateAndSave();
		} catch (Exception e) {
			warning("processValues", "Unable to save site identifier for object id " + resId + " and type ID " + idtId + " with value " + value, e);
		}
	}

	protected String createAccessPoint(String resId, String acpType, String adapterName) throws Exception {
		try {
			if (!adapterName.equals("SitesRaftAD")) {
				List<String[]> rows = acpTool.search(new JSONObject().put("sitesAcpReaId", resId).put("sitesAcpType", acpType));
				if (rows.size() != 0) { // Create if not exists (no need to update)
					return acp.getRowId();
				}
			}
			acp.resetValues(true);
			acp.setFieldValue("sitesAcpReaId", resId);
			acp.setFieldValue("sitesAcpType", acpType);
			acp.setFieldValue("sitesGenStatus", "ACTIVE");
			acpTool.validateAndSave();
			return acp.getRowId();
		} catch (Exception e) {
			throw new Exception("Unable to save access point for site ID " + resId);
		}
	}

	protected void createOrUpdateAccessPointIdentifier(String identifierType, String acpId, String value, String sourceSystem) throws Exception {
		String idtId = getIdentifierTypeId(identifierType);
		ArrayList<String> existingAcpIds = acp.getRowIdsFromAlternativeIdentifier(value, identifierType, false);
		// We create only one RAFT_CODE business identifier for each pair type/value of an Access Point
		if (!existingAcpIds.contains(acpId)) {
			if (Tool.isEmpty(idtId)) {
				throw new Exception("Configuration issue:" + identifierType + " is not set up");
			}
			try {
				api.resetValues(true);
				api.setFieldValue("sitesApiAcpId", acpId);
				api.setFieldValue("sitesApiIdtId", idtId);
				api.setFieldValue("sitesApiSysId", getSourceSystemId(sourceSystem));
				api.setFieldValue("sitesApiValue", value);
				if (debug) {
					appendLog(new JSONObject(api.toJSON()).toString(2));
				}
				apiTool.validateAndSave();
			} catch (Exception e) {
				throw new Exception("Unable to save access point identifier for object id " + acpId + " and type ID " + idtId + " with value " + value);
			}
		}
	}

	protected String createOrUpdateCity(String ctyId, String postalCode, String cityName, String countryCode, String resId) {
		String citId = null;
		if (cityName != null) {
			citId = getCityId(countryCode, cityName);
			if (citId == null) {
				try {
					cit.resetValues(true);
					cit.setFieldValue("sitesCitCtyId", ctyId);
					cit.setFieldValue("sitesCitName", cityName);
					if (!postalCode.equals("")) {
						cit.setFieldValue("sitesCitPostalCode", postalCode);
					}
					if (debug) {
						appendLog(new JSONObject(cit.toJSON()).toString(2));
					}
					citTool.validateAndSave();
					citId = cit.getRowId();
				} catch (Exception e) {
					warning("processValues", "Warning: Unable to create city", e);
				}
			}

			// Update site's city if needed
			if (citId != null) {
				res.resetFilters();
				res.select(resId);
				ObjectField sitesResCitId = res.getField("sitesResCitId");
				if (!citId.equals(sitesResCitId.getValue())) try {
					sitesResCitId.setValue(citId);
					resTool.validateAndSave();
				} catch (Exception e) {
					warning("processValues", "Unable to update site ID " + resId + " with city ID " + citId, null);
				}
			}
		}

		if (citId == null) {
			citId = "";
		}
		return citId;
	}

	protected void createAccessPointAddress(String acpId, String adrId, String adrType, String adrSummary, String adapterName) throws Exception {
		try {
			// If adapterName = SitesRaftAD we don't need to search for existing acpAdr because the known process flow guarantees that there are no acpAdr 
			if (!adapterName.equals("SitesRaftAD")) {
				List<String[]> rows = acpAdrTool.search(new JSONObject().put("sitesAcpAdrAcpId", acpId).put("sitesAcpAdrAdrId", adrId)); 
				if (rows.size() != 0) { // Link for specified access point id and address id already existing so no need to update
					return;
				}
			}
			// If we're being called by SitesRaftAD or in any other call for a non existing acpAdr between adrId and acpId
			acpAdr.resetValues(true);
			acpAdr.setFieldValue("sitesAcpAdrAcpId", acpId);
			acpAdr.setFieldValue("sitesAcpAdrAdrId", adrId);
			acpAdr.setFieldValue("sitesAdrType", adrType);
			acpAdr.setFieldValue("sitesAdrSummary", adrSummary);
			acpAdrTool.validateAndSave();
		} catch (Exception e) {
			throw new Exception("Unable to save access point address for acpId: " + acpId + ", adrId: " + adrId + ", adrType: " + adrType + ", adrSummary: " + adrSummary);
		}
	}

	protected void resetAccessPointAddresses(String acpId) throws Exception {
		try {
			List<String[]> rows = acpAdrTool.search(new JSONObject().put("sitesAcpAdrAcpId", acpId));
			for (int i = 0; i < rows.size(); i++) {
				acpAdr.setValues(rows.get(i));
				acpAdrTool.delete();
			}
		} catch (Exception e) {
			throw new Exception("Unable to reset access point address for acpId: " + acpId);
		}
	}	

	@Override
	public void postProcess() {
		super.postProcess();

		appendLog("=================================================================");
		appendLog("Nb lines processed = " + nbLines);
		appendLog("Nb warnings = " + nbWarnings);
		appendLog("Nb errors = " + nbErrors);
	}
}
1 Like

En regardant rapidement la structure générale vote code je ne vois pas de raison évidente pour que le traitement plante méchamment en plein milieu : il y a des try/catch à priori bien placés pour protéger la poursuite du traitement.

Je vous recommande donc de mettre un maximum de traces pour cerner l’endroit précis où le traitement s’arrête et/ou sur quelles données (peut être un pb d’encoding ou dans le genre qui provoque une exception/throwable non prévue)

A mon humble avis ce n’est lié ni au code ni aux données pour ces raisons:

j’ai utilisé le meme code et les memes données pour un import que j’ai effectué en janvier. Au début, lors du lancement de l’adapter, des null pointer apparaissaient dans les logs qu’on arrivait pas à expliquer. suite à votre recommandation (que c’est un soucis lié au démarrage de l’import), j’ai effectué des imports progressives (10lignes puis 100 puis 1000 puis 5000 …) et l’import s’est terminé sans soucis.
cependant, après le passage à la P24, j’ai l’impression que cet import ne s’effectue plus. je n’ai plus les nulls pointers dans les logs mais l’import s’arrete sans erreur visible en tout cas.
actuellement je fais un import par lot de 2000 lignes mais ce n’est pas trop pratique

Je vais laisser @Francois répondre car je sais qu’il y a eu des évolutions sur la manière dont les flux entrants sont gérés (stream vs mémoire), je ne sais pas exactement ce qui a été intégré dans le cadre de la release P24 vs ce qui a été fait uniquement sur la version de dev actuelle.

PS: Je précise quand même que votre P24 d’il y a 1 mois n’est pas à jour de plus de 40 commits vis à vis de la P24 actuelle.

oui je vous ai envoyé le mauvais /health
voici l’actuel:
[Platform]
Status=OK
Version=4.0.P24
BuiltOn=2020-04-01 19:15 (revision 08ba761b0feac78eab1f12fa057ebb440186e282)
Encoding=UTF-8
EndpointIP=172.17.0.4
EndpointURL=http://8ea229b517c0:8080
TimeZone=Europe/Paris
SystemDate=2020-04-02 15:26:59

[Application]
ApplicationVersion=0.16 dev
ContextPath=
ContextURL=https://ope.rfs.ope.aws.renault.com
ActiveSessions=2
TotalUsers=9
EnabledUsers=2
LastLoginDate=

OK, vous êtes à jour.

Je ne vois rien de bizarre dans le code sauf sur les instances “tmp” qui ne sont pas synchronisées
(SitesAccessPoint) getGrant().getTmpObject(“SitesAccessPoint”);

En général on instancie des getBatchObject (instance dédiée aux imports) ou on met des synchronized(obj) pour éviter qu’un autre hook ou import l’utilise pour rendre atomique les resetFilter+search/select+update de chaque instance.

Le fichier importé ne monte plus en mémoire avant d’être traité, les données sont lues par buffer (type readLine ou import SAX…).

Comment l’import est il appelé ?
que fait la classe RenaultSites.SitesCommonAdapter ?

Ne serait ce pas juste un pb de timeout de call HTTP synchrone ?
ou quelque chose qui stoppe le flux entrant ?
Il devrait y a voir une trace quelque part de type IOException (log de l’adapter, catalina de tomcat, simplicite.log ?).

voici comment l’import se fait:

la classe siteCommon renferme des méthodes qui sont utilisées dans plusieurs adapters

voici les paramètres timeout :

dois je changer? des propositions.
je n’ai pas de logs qui s’enregistre car le fichier log devait se générer à la fin de l’import

Je vais faire des tests avec un gros fichier et un timeout à 10 secondes.
Essayez de mettre 0 au lieu de 30 le temps d’un test de votre côté.

Test OK de mon côté en passant par cette interface de la UI P24 :

  • import de 72000 lignes (création de commandes de la démo)
  • via l’adapter CSVAdapter en mode direct en 30 minutes environ
  • log normales avec des rejets normaux de lignes invalides
  • sans rien toucher par ailleurs (pas de clear cache, pas de navigation dans cette session Chrome), juste en regardant depuis une autre session (firefox) la liste des commandes créées

Fonctionnement de cette UI :

  • commence par charger le fichier en back dans le répertoire WEB-INF/tmp avec un hash unique, c’est là qu’il faut vérifier que tout votre flux arrive bien à destination
  • Ensuite un reader de l’adapter parcourt le fichier ligne par ligne, il n’y a pas de timeout à ce thread, mais il faut veiller à ce qu’il soit thread safe (cf remarque sur vos instances non dédiées / non synchronisées)
  • Ensuite dans WEB-INF/tmp vous pouvez suivre les fichiers appender log + err (tail -f)
  • A la fin, ils sont placés en base dans la supervision des imports et supprimés de tmp.
  • Si tmp n’est pas vidé c’est que le thread/import n’a pas terminé correctement

Vous avez peut être une exception non reconnue : mettez plutôt un catch(Throwable e) qui capte les erreurs de plus bas niveau que Exception.

Dernier point, en combien de temps vous chargez votre fichier (en interpolant avec ce que vous chargez déjà) ?

Imaginons un programme qui prend 1h, puis à la fin utilise un objet préparé au tout début, il ne sera plus en mémoire, et ça peut générer des comportements non prévus.

Car si un objet n’est plus utilisé après 15 minutes d’inactivité, le garbage collector des cron Simplicité le détruit (les données chargées après 15min, puis l’instance de la mémoire après 30min). Il convient de refaire les getBatchObject dans votre boucle avant de les utiliser, pour les réinstancier au cas où.

En regardant de plus près les logs tomcat, il y a effectivement une erreur qui doit correspondre à votre problème de stockage du résultat :

2020-04-02 17:47:57,663|SIMPLICITE|http://DESKTOP-EHE6EO8:9080/simplicite|/simplicite|ECORED0001|system|com.simplicite.util.IntegrationDirect|importADP||Error Problem in XML import thru adapter CSVAdapter
java.lang.NullPointerException
	at com.simplicite.util.ObjectDB.select(ObjectDB.java:792)
	at com.simplicite.util.ObjectDB.select(ObjectDB.java:829)
	at com.simplicite.objects.System.XMLSupervisor.storeAdapterResult(XMLSupervisor.java:93)
        ...

Je vais creuser cette piste… mais mes remarques restent applicables pour sécuriser votre code.

Bonjour,
Oui effectivement c’est ces null pointeur qui s’empilent dans mes logs quand j’essaie d’importer d’un coup
Merci!

Merci, on va pousser une correction de ce problème ce soir.
Par contre ça n’a rien à voir avec le fait que le flux ne soit pas totalement importé de votre côté.

Dans mon test 72000 lignes ont bien été importées ou rejetées, c’est le storage des logs à la fin qui a eu ce problème.

Bonjour,
Le fait de changer le HTTPVALL_TIMEOUT à 0 semble régler le problème. mon import est en cours pour le moment, pourvu que ça dure.
Pour info, l’architecture du modèle est un peu compliqué (c’est des données pour créer des sites, des access points pour chaque site, des adresses et des business identifers pour chaque access point…).
Cependant, j’ai redéployé mon instance ce matin, et j’ai ces messages d’erreur :

2020-04-03 09:01:25,699 ERROR [com.simplicite.util.engine.ObjectManager] SIMPLICITE|http://d2647e4a7be5:8080||ERROR|system|com.simplicite.util.engine.ObjectManager|makeFrom||Event: Configuration error SocialPost: unable to find the parent field of pstUserId.usr_image_id
2020-04-03 09:01:25,699 ERROR [com.simplicite.util.engine.ObjectManager] SIMPLICITE|http://d2647e4a7be5:8080||ERROR|system|com.simplicite.util.engine.ObjectManager|makeFrom||Event: Configuration error SocialPost: unable to find the parent field of pstUserId.usr_email
2020-04-03 09:01:25,699 ERROR [com.simplicite.util.engine.ObjectManager] SIMPLICITE|http://d2647e4a7be5:8080||ERROR|system|com.simplicite.util.engine.ObjectManager|makeFrom||Event: Configuration error SocialPost: unable to find the parent field of pstUserId.usr_last_name
2020-04-03 09:01:25,699 ERROR [com.simplicite.util.engine.ObjectManager] SIMPLICITE|http://d2647e4a7be5:8080||ERROR|system|com.simplicite.util.engine.ObjectManager|makeFrom||Event: Configuration error SocialPost: unable to find the parent field of pstUserId.usr_first_name
2020-04-03 09:01:25,699 ERROR [com.simplicite.util.engine.ObjectManager] SIMPLICITE|http://d2647e4a7be5:8080||ERROR|system|com.simplicite.util.engine.ObjectManager|makeFrom||Event: Configuration error SocialPost: unable to find the parent field of pstUserId.usr_login
2020-04-03 09:01:25,292 ERROR [com.simplicite.util.Grant] SIMPLICITE|http://d2647e4a7be5:8080||ERROR|system|com.simplicite.util.Grant|updateMetaObjects||Event: Unable to update the meta-object: ALMTestCase.ALMTestCaseObject
    java.lang.NullPointerException
     at com.simplicite.util.Grant.updateMetaObjects(Grant.java:2674)
     at com.simplicite.util.engine.ObjectManager.update(ObjectManager.java:2576)
     at com.simplicite.util.engine.ObjectManager.save(ObjectManager.java:2927)
     at com.simplicite.util.ObjectDirect.save(ObjectDirect.java:441)
     at com.simplicite.util.ObjectDB.save(ObjectDB.java:1135)
     at com.simplicite.util.ObjectDB.save(ObjectDB.java:1122)
     at com.simplicite.util.tools.IndexSQLTool.indexFields(IndexSQLTool.java:205)
     at com.simplicite.util.tools.IndexSQLTool.indexation(IndexSQLTool.java:254)
     at com.simplicite.util.engine.ObjectManager.indexObjectSync(ObjectManager.java:4385)
     at com.simplicite.util.engine.ObjectManager$1.run(ObjectManager.java:4422)
     at java.base/java.lang.Thread.run(Thread.java:830)
2020-04-03 09:01:25,291 ERROR [com.simplicite.util.engine.CoreCache] SIMPLICITE|http://d2647e4a7be5:8080||ECOREGT002|system|com.simplicite.util.engine.CoreCache|instantiateObject||Fix the class or object ALMTestCase
    java.lang.NullPointerException
     at com.simplicite.util.engine.CoreCache.instantiateObject(CoreCache.java:3588)
     at com.simplicite.util.engine.CoreCache.getObject(CoreCache.java:3542)
     at com.simplicite.util.engine.GrantManager.getObject(GrantManager.java:96)
     at com.simplicite.util.GrantDirect.getObject(GrantDirect.java:571)
     at com.simplicite.util.Grant.getObject(Grant.java:1866)
     at com.simplicite.util.Grant.updateMetaObjects(Grant.java:2673)
     at com.simplicite.util.engine.ObjectManager.update(ObjectManager.java:2576)
     at com.simplicite.util.engine.ObjectManager.save(ObjectManager.java:2927)
     at com.simplicite.util.ObjectDirect.save(ObjectDirect.java:441)
     at com.simplicite.util.ObjectDB.save(ObjectDB.java:1135)
     at com.simplicite.util.ObjectDB.save(ObjectDB.java:1122)
     at com.simplicite.util.tools.IndexSQLTool.indexFields(IndexSQLTool.java:205)
     at com.simplicite.util.tools.IndexSQLTool.indexation(IndexSQLTool.java:254)
     at com.simplicite.util.engine.ObjectManager.indexObjectSync(ObjectManager.java:4385)
     at com.simplicite.util.engine.ObjectManager$1.run(ObjectManager.java:4422)
     at java.base/java.lang.Thread.run(Thread.java:830)
2020-04-03 09:01:25,291 FATAL [com.simplicite.util.ObjectDirect] SIMPLICITE|http://d2647e4a7be5:8080||FCOREOM002|system|com.simplicite.util.ObjectDirect|init||Error init Object ALMTestCase for user system
    java.lang.NullPointerException
     at com.simplicite.util.ObjectCore.getModuleName(ObjectCore.java:339)
     at com.simplicite.util.engine.ObjectLoader.loadFromDB(ObjectLoader.java:503)
     at com.simplicite.util.engine.ObjectLoader.getClone(ObjectLoader.java:131)
     at com.simplicite.util.engine.ObjectLoader.load(ObjectLoader.java:87)
     at com.simplicite.util.ObjectDirect.init(ObjectDirect.java:41)
     at com.simplicite.util.ObjectDB.init(ObjectDB.java:237)
     at com.simplicite.util.ObjectDB.load(ObjectDB.java:215)
     at com.simplicite.util.engine.CoreCache.instantiateObject(CoreCache.java:3583)
     at com.simplicite.util.engine.CoreCache.getObject(CoreCache.java:3542)
     at com.simplicite.util.engine.GrantManager.getObject(GrantManager.java:96)
     at com.simplicite.util.GrantDirect.getObject(GrantDirect.java:571)
     at com.simplicite.util.Grant.getObject(Grant.java:1866)
     at com.simplicite.util.Grant.updateMetaObjects(Grant.java:2673)
     at com.simplicite.util.engine.ObjectManager.update(ObjectManager.java:2576)
     at com.simplicite.util.engine.ObjectManager.save(ObjectManager.java:2927)
     at com.simplicite.util.ObjectDirect.save(ObjectDirect.java:441)
     at com.simplicite.util.ObjectDB.save(ObjectDB.java:1135)
     at com.simplicite.util.ObjectDB.save(ObjectDB.java:1122)
     at com.simplicite.util.tools.IndexSQLTool.indexFields(IndexSQLTool.java:205)
     at com.simplicite.util.tools.IndexSQLTool.indexation(IndexSQLTool.java:254)
     at com.simplicite.util.engine.ObjectManager.indexObjectSync(ObjectManager.java:4385)
     at com.simplicite.util.engine.ObjectManager$1.run(ObjectManager.java:4422)
     at java.base/java.lang.Thread.run(Thread.java:830)
2020-04-03 09:01:25,288 ERROR [com.simplicite.util.Grant] SIMPLICITE|http://d2647e4a7be5:8080||ERROR|system|com.simplicite.util.Grant|updateMetaObjects||Event: Unable to update the meta-object: ALMImplementation.ALMImplObject
    java.lang.NullPointerException
     at com.simplicite.util.Grant.updateMetaObjects(Grant.java:2674)
     at com.simplicite.util.engine.ObjectManager.update(ObjectManager.java:2576)
     at com.simplicite.util.engine.ObjectManager.save(ObjectManager.java:2927)
     at com.simplicite.util.ObjectDirect.save(ObjectDirect.java:441)
     at com.simplicite.util.ObjectDB.save(ObjectDB.java:1135)
     at com.simplicite.util.ObjectDB.save(ObjectDB.java:1122)
     at com.simplicite.util.tools.IndexSQLTool.indexFields(IndexSQLTool.java:205)
     at com.simplicite.util.tools.IndexSQLTool.indexation(IndexSQLTool.java:254)
     at com.simplicite.util.engine.ObjectManager.indexObjectSync(ObjectManager.java:4385)
     at com.simplicite.util.engine.ObjectManager$1.run(ObjectManager.java:4422)
     at java.base/java.lang.Thread.run(Thread.java:830)
2020-04-03 09:01:25,285 ERROR [com.simplicite.util.engine.CoreCache] SIMPLICITE|http://d2647e4a7be5:8080||ECOREGT002|system|com.simplicite.util.engine.CoreCache|instantiateObject||Fix the class or object ALMImplementation
    java.lang.NullPointerException
     at com.simplicite.util.ObjectCore.getScriptId(ObjectCore.java:363)
     at com.simplicite.util.engine.CoreCache.instantiateObject(CoreCache.java:3588)
     at com.simplicite.util.engine.CoreCache.getObject(CoreCache.java:3542)
     at com.simplicite.util.engine.GrantManager.getObject(GrantManager.java:96)
     at com.simplicite.util.GrantDirect.getObject(GrantDirect.java:571)
     at com.simplicite.util.Grant.getObject(Grant.java:1866)
     at com.simplicite.util.Grant.updateMetaObjects(Grant.java:2673)
     at com.simplicite.util.engine.ObjectManager.update(ObjectManager.java:2576)
     at com.simplicite.util.engine.ObjectManager.save(ObjectManager.java:2927)
     at com.simplicite.util.ObjectDirect.save(ObjectDirect.java:441)
     at com.simplicite.util.ObjectDB.save(ObjectDB.java:1135)
     at com.simplicite.util.ObjectDB.save(ObjectDB.java:1122)
     at com.simplicite.util.tools.IndexSQLTool.indexFields(IndexSQLTool.java:205)
     at com.simplicite.util.tools.IndexSQLTool.indexation(IndexSQLTool.java:254)
     at com.simplicite.util.engine.ObjectManager.indexObjectSync(ObjectManager.java:4385)
     at com.simplicite.util.engine.ObjectManager$1.run(ObjectManager.java:4422)
     at java.base/java.lang.Thread.run(Thread.java:830)
2020-04-03 09:01:24,968 ERROR [com.simplicite.util.engine.ObjectManager] SIMPLICITE|http://d2647e4a7be5:8080||ERROR|system|com.simplicite.util.engine.ObjectManager|makeFrom||Event: Configuration error SocialPost: unable to find the parent field of pstUserId.usr_image_id
2020-04-03 09:01:24,968 ERROR [com.simplicite.util.engine.ObjectManager] SIMPLICITE|http://d2647e4a7be5:8080||ERROR|system|com.simplicite.util.engine.ObjectManager|makeFrom||Event: Configuration error SocialPost: unable to find the parent field of pstUserId.usr_email
2020-04-03 09:01:24,968 ERROR [com.simplicite.util.engine.ObjectManager] SIMPLICITE|http://d2647e4a7be5:8080||ERROR|system|com.simplicite.util.engine.ObjectManager|makeFrom||Event: Configuration error SocialPost: unable to find the parent field of pstUserId.usr_last_name
2020-04-03 09:01:24,968 ERROR [com.simplicite.util.engine.ObjectManager] SIMPLICITE|http://d2647e4a7be5:8080||ERROR|system|com.simplicite.util.engine.ObjectManager|makeFrom||Event: Configuration error SocialPost: unable to find the parent field of pstUserId.usr_first_name
2020-04-03 09:01:24,968 ERROR [com.simplicite.util.engine.ObjectManager] SIMPLICITE|http://d2647e4a7be5:8080||ERROR|system|com.simplicite.util.engine.ObjectManager|makeFrom||Event: Configuration error SocialPost: unable to find the parent field of pstUserId.usr_login
2020-04-03 09:01:24,938 ERROR [com.simplicite.util.engine.CoreCache] SIMPLICITE|http://d2647e4a7be5:8080||ERROR|system|com.simplicite.util.engine.CoreCache|getField||Event: Language ENU not found for field pstObject
2020-04-03 09:01:24,937 ERROR [com.simplicite.util.engine.CoreCache] SIMPLICITE|http://d2647e4a7be5:8080||ERROR|system|com.simplicite.util.engine.CoreCache|getField||Event: Language ENU not found for field scu_object
2020-04-03 09:01:24,936 ERROR [com.simplicite.util.engine.CoreCache] SIMPLICITE|http://d2647e4a7be5:8080||ERROR|system|com.simplicite.util.engine.CoreCache|getField||Event: Language ENU not found for field res_object
2020-04-03 09:01:24,935 ERROR [com.simplicite.util.engine.CoreCache] SIMPLICITE|http://d2647e4a7be5:8080||ERROR|system|com.simplicite.util.engine.CoreCache|getField||Event: Language ENU not found for field res_object
2020-04-03 09:01:24,934 ERROR [com.simplicite.util.engine.CoreCache] SIMPLICITE|http://d2647e4a7be5:8080||ERROR|system|com.simplicite.util.engine.CoreCache|getField||Event: Language ENU not found for field rlg_object
2020-04-03 09:01:24,934 ERROR [com.simplicite.util.engine.CoreCache] SIMPLICITE|http://d2647e4a7be5:8080||ERROR|system|com.simplicite.util.engine.CoreCache|getField||Event: Language ENU not found for field mdi_target
2020-04-03 09:01:24,933 ERROR [com.simplicite.util.engine.CoreCache] SIMPLICITE|http://d2647e4a7be5:8080||ERROR|system|com.simplicite.util.engine.CoreCache|getField||Event: Language ENU not found for field csi_target
2020-04-03 09:01:24,932 ERROR [com.simplicite.util.engine.CoreCache] SIMPLICITE|http://d2647e4a7be5:8080||ERROR|system|com.simplicite.util.engine.CoreCache|getField||Event: Language ENU not found for field pcf_object
2020-04-03 09:01:24,931 ERROR [com.simplicite.util.engine.CoreCache] SIMPLICITE|http://d2647e4a7be5:8080||ERROR|system|com.simplicite.util.engine.CoreCache|getField||Event: Language ENU not found for field job_object_inst
2020-04-03 09:01:24,930 ERROR [com.simplicite.util.engine.CoreCache] SIMPLICITE|http://d2647e4a7be5:8080||ERROR|system|com.simplicite.util.engine.CoreCache|getField||Event: Language ENU not found for field ALMTestCaseObject
2020-04-03 09:01:24,929 ERROR [com.simplicite.util.engine.CoreCache] SIMPLICITE|http://d2647e4a7be5:8080||ERROR|system|com.simplicite.util.engine.CoreCache|getField||Event: Language ENU not found for field ALMImplObject
2020-04-03 09:01:24,924 ERROR [com.simplicite.util.engine.CoreCache] SIMPLICITE|http://d2647e4a7be5:8080||ERROR|system|com.simplicite.util.engine.CoreCache|getField||Event: Language ENU not found for field pstObject
2020-04-03 09:01:24,923 ERROR [com.simplicite.util.engine.CoreCache] SIMPLICITE|http://d2647e4a7be5:8080||ERROR|system|com.simplicite.util.engine.CoreCache|getField||Event: Language ENU not found for field pstLevel
2020-04-03 09:01:24,922 ERROR [com.simplicite.util.engine.CoreCache] SIMPLICITE|http://d2647e4a7be5:8080||ERROR|system|com.simplicite.util.engine.CoreCache|getField||Event: Language ENU not found for field pstStatus
2020-04-03 09:01:24,921 ERROR [com.simplicite.util.engine.CoreCache] SIMPLICITE|http://d2647e4a7be5:8080||ERROR|system|com.simplicite.util.engine.CoreCache|getField||Event: Language ENU not found for field pstFollowersOnly
2020-04-03 09:01:24,920 ERROR [com.simplicite.util.engine.CoreCache] SIMPLICITE|http://d2647e4a7be5:8080||ERROR|system|com.simplicite.util.engine.CoreCache|getField||Event: Language ENU not found for field pstMessage
2020-04-03 09:01:24,919 ERROR [com.simplicite.util.engine.CoreCache] SIMPLICITE|http://d2647e4a7be5:8080||ERROR|system|com.simplicite.util.engine.CoreCache|getField||Event: Language ENU not found for field pstDate
2020-04-03 09:01:24,918 ERROR [com.simplicite.util.engine.CoreCache] SIMPLICITE|http://d2647e4a7be5:8080||ERROR|system|com.simplicite.util.engine.CoreCache|getField||Event: Language ENU not found for field usr_image_id
2020-04-03 09:01:24,917 ERROR [com.simplicite.util.engine.CoreCache] SIMPLICITE|http://d2647e4a7be5:8080||ERROR|system|com.simplicite.util.engine.CoreCache|getField||Event: Language ENU not found for field usr_email
2020-04-03 09:01:24,916 ERROR [com.simplicite.util.engine.CoreCache] SIMPLICITE|http://d2647e4a7be5:8080||ERROR|system|com.simplicite.util.engine.CoreCache|getField||Event: Language ENU not found for field usr_last_name
2020-04-03 09:01:24,916 ERROR [com.simplicite.util.engine.CoreCache] SIMPLICITE|http://d2647e4a7be5:8080||ERROR|system|com.simplicite.util.engine.CoreCache|getField||Event: Language ENU not found for field usr_first_name
2020-04-03 09:01:24,915 ERROR [com.simplicite.util.engine.CoreCache] SIMPLICITE|http://d2647e4a7be5:8080||ERROR|system|com.simplicite.util.engine.CoreCache|getField||Event: Language ENU not found for field usr_login
2020-04-03 09:01:24,914 ERROR [com.simplicite.util.engine.CoreCache] SIMPLICITE|http://d2647e4a7be5:8080||ERROR|system|com.simplicite.util.engine.CoreCache|getField||Event: Language ENU not found for field pstUserId
2020-04-03 09:01:24,907 ERROR [com.simplicite.util.engine.CoreCache] SIMPLICITE|http://d2647e4a7be5:8080||ERROR|system|com.simplicite.util.engine.CoreCache|getField||Event: Language ENU not found for field updated_by
2020-04-03 09:01:24,906 ERROR [com.simplicite.util.engine.CoreCache] SIMPLICITE|http://d2647e4a7be5:8080||ERROR|system|com.simplicite.util.engine.CoreCache|getField||Event: Language ENU not found for field updated_dt
2020-04-03 09:01:24,905 ERROR [com.simplicite.util.engine.CoreCache] SIMPLICITE|http://d2647e4a7be5:8080||ERROR|system|com.simplicite.util.engine.CoreCache|getField||Event: Language ENU not found for field created_by
2020-04-03 09:01:24,903 ERROR [com.simplicite.util.engine.CoreCache] SIMPLICITE|http://d2647e4a7be5:8080||ERROR|system|com.simplicite.util.engine.CoreCache|getField||Event: Language ENU not found for field created_dt
2020-04-03 09:01:24,902 ERROR [com.simplicite.util.engine.CoreCache] SIMPLICITE|http://d2647e4a7be5:8080||ERROR|system|com.simplicite.util.engine.CoreCache|getField||Event: Language ENU not found for field row_id
2020-04-03 09:01:24,898 ERROR [com.simplicite.util.engine.CoreCache] SIMPLICITE|http://d2647e4a7be5:8080||ERROR|system|com.simplicite.util.engine.CoreCache|getField||Event: Language ENU not found for field idx_key

est ce lié à mon import?

J’ai fait un test avec un timeout à 30 secondes sans soucis alors que l’import a mis 30 minutes.
Après vérification, ce paramètre système ne sert pas sur cet appel, il est utilisé lorsque le serveur appelle des web services externes de façon synchrone (cf tous les Tool.readUrl…).

Les paramètres de timeout sont donc ceux de votre connector http, il faut vérifier dans votre server.xml de tomcat le connector HTTP s’il n’a pas une limitation quelconque : connectionTimeout, connectionUploadTimeout, etc.

De mon côté, il y a juste un connectionTimeout=20000 (20 secondes) qui n’affecte pas l’envoi de fichier ni son traitement.

Il faut vérifier quand vous avez un import partiel si le fichier a bien été transmis depuis le navigateur dans WEB-INF/tmp ou depuis la supervision des imports, téléchargez la source réellement reçue :

Aussi regardez depuis le navigateur l’appel à io?service=IMPORTADP dans le debugger Network :

En cours :

Et à la fin :

  • La requête a pris 140 ms (donc bien inférieur 20 secondes)
  • et le traitement 30 minutes

Sinon le problème de SocialPost (des rapports d’audit j’imagine sans incidence) est déclenché par des objets ALM. Il y a dû y avoir un problème d’accès concurrent entre un vider le cache et un upgrade en cours. Il faut vider vos caches mais ça n’a rien à voir avec votre import.

Merci François pour ces explications.
Je trouve quand même bizarre que le fait d’avoir changé uniquement le HTTPCALL a impacté l’import (dans le bon sens) ou ça a peut être simplement coïncidé par hasard.
En tout cas, pour le moment l’import se poursuit correctement (je suis à 75% du fichier et c’est normal que ça prenne plus de temps) et je n’ai pas fait de changement depuis hier.

Re Bonjour,
L’import vient de s’arrêter avant la fin (il reste 20 lignes seulement cette fois heureusement).
j’ai vérifié la source (fichier complet) et dans les logs de la console

à votre avis?