👷Files (WIP)

ONLY use the file connector included in the edge-connector to import and manage files on your own devices. The file connector under data connectors in the app builder does not have file permanence and should not be used.

The Files connector can read specific file types, write .csv files, and list folder contents. It can also move, copy, or delete files. The connector operates on the Heisenware server in your files directory. To access files on your own device, please go to the edge-connector and return here, once you have successfully deployed it.

The path the file connector is operating in is your starting point for all the path input fields. To see the contents of the root folder and all subfolders, use the browse function and insert a . into the path input box.

To go up a level in the folder structure, use .. in the beginning of the input field.

Read a file

The readFile function is a cumulative function of the others, currently supporting .csv, .xlsx, .txt, .pdf and .xml file types. Simply insert the path relative to the root folder into the input box. Parameters can be added in object format parameter: value. For possible parameters of each file type see below.

Read a .csv file

To read a character separated values-file use the readCsv function and insert the path into the input box. Parameters can be added in object format in the second box.

Parameter	Default	Description
delimiter	,	Delimiter used for separating columns. Use "auto" if delimiter is unknown in advance. Use an array to give a list of potential delimiters e.g. [", " "|", "$"].
noheader	false	Indicating csv data has no header row and first row is data row.
checkColumn	false	Check whether column number of a row is the same as headers. If column number mismatched headers number, an error of "mismatched_column" will be emitted.
checkType	false	Turns on and off type interpretation
quote	"	If a column contains delimiter, it is able to use quote character to surround the column content. e.g. "hello, world" won't be split into two columns while parsing. Set to "off" will ignore all quotes.
trim	true	Indicate if parser trim off spaces surrounding column content. e.g. " content " will be trimmed to "content".
ignoreEmpty	false	Ignore the empty value in CSV columns. If a column value is not given, set this to true to skip them.
includeColumns		This parameter instructs the parser to include only those columns as specified by the regular expression. Example: /(name|age)/ will parse and include columns whose header contains "name" or "age"
ignoreColumns		This parameter instructs the parser to ignore columns as specified by the regular expression. Example: /(name|age)/ will ignore columns whose header contains "name" or "age"

See https://www.npmjs.com/package/csvtojson#parameters for more parameter options.

Write a .csv file

You can write .csv files with the writeCsv function. Simply insert or link some JSON string in the first input box and specify a path and filename for the file to be saved to. The delimiter of the values will always be a comma.

Parameters from the readCSV function can currently NOT be used for writing .csv files.

Read an Excel sheet (.xlsx)

Use the readXlsx function to ingest Excel files.

Output Example:

{
    sheet1: [{
        A: 'data of cell A1',
        B: 'data of cell B1',
        C: 'data of cell C1'
    },
    {
        A: 'data of cell A2',
        B: 'data of cell B2',
        C: 'data of cell C2'
    }],
    sheet2: [{
        A: 'data of cell A1',
        B: 'data of cell B1',
        C: 'data of cell C1'
    }]
}

Parameter	Example	Description
header	{rows: 1}	This is the number of rows that will be skipped and will not be present in the resulting object. Counting from top to bottom.
sheets	['sheet2']	Only get the data from a specific sheet.
columnToKey	static example: { A: 'id', B: 'firstName' } dynamic example: { '*': '{{columnHeader}}' }	Name columns in the output. It is possible to use a value from the sheet with e.g. '{{A1}}' or {{columnHeader}}, which will follow the header parameter settings. To dynamically name every column, use '*'. Omitting columns ignores them for the output, even if specified in the range parameter.
range	'A2:B3'	Defines the range from which to include data. If your column range goes into double characters (e.g.: AG), set the range from A until Z, because the submodule currently works with alphabetical sorting internally.
sheetStubs	false	To include empty cells (NULL values), set this to true.

It is also possible to nest most parameters for different options per sheet. See https://github.com/DiegoZoracKy/convert-excel-to-json for all parameter options and examples.

Read a Word file (.docx)

The readDocx function is working with textract in the background. You can ingest Word documents with it.

Parameter	Default	Description
preserveLineBreaks	false	Pass this in as true and textract will not strip any line breaks. Line breaks are preserved most of the time even with false, but to make sure to preserve line breaks, set this to true.
preserveOnlyMultipleLineBreaks	false	Some extractors, like PDF, insert line breaks at the end of every line, even in the middle of a sentence. If this option is set to true, then any instances of a single line break are removed but multiple line breaks are preserved. Check your output with this option, though, as this doesn't preserve paragraphs unless there are multiple breaks.

For more options visit https://www.npmjs.com/package/textract#configuration.

The following functions rely on the same textract component, but for different file types different parameters might be sensible.

Read a PowerPoint file (.pptx)

The readPptx function allows you to ingest PowerPoint files and also works with textract. For parameter options see Read a Word file.

Read an HTML file (.html & .htm)

The readHtml function allows for reading of HTML files in .html and .htm format. It also works with textract, so it has the same parameter options as the function for Word files.

Additionally, there is includeAltText: When extracting HTML, whether or not to include alt text with the extracted text. By default this is false.

Read a text file (.txt)

Reading a text file is also textract based (see Word file) and can be done with the function readTxt.

Read a markdown file (.md)

The function readMd is also textract based (see Word file) and can be used to ingest markdown files.

Read a PDF file

PDFs can be read with the function readPdf, which is also textract based like the Word function. Additionally to the parameters explained above, for PDF functions there is also pdftotextOptions. Parameters entered here need to be nested with pdftotextOptions being the main key and all sub options being nested in the input object like pdftotextOptions: {ownerPassword: 123}.

pdftotextOptions: This is a proxy options object to the library textract uses for pdf extraction: pdf-text-extract. Options include ownerPassword, userPassword if you are extracting text from password protected PDFs.

Textract modifies the pdf-text-extract layout default so that, instead of layout: layout, it uses layout:raw. Do not modify this without understanding what problems might arise. See this GH issue for why textract overrides that library's default.

Read a .xml file

To read extended markup language files use the readXml function. It is also textract based and therefore offers the same parameters as the Word function.

Move a file

Use the moveFile function to remove a file from one location and insert it in a new one. Simply insert the old path, including the file name, in the first input box and the new path, also inclusive of file name, in the second input box. Moving a file into another folder is only possible if it already exists.

Moving a file from the root folder into a subfolder while keeping the name.

Copy a file

With the copyFile function, you can duplicate files and simultaneously move the copy to a new location.

Duplicating a file into the same folder.

Delete a file

To delete a file, use the deleteFile function.

Deleting the file2.txt file.

Files get deleted directly and cannot be recovered!

Create a new folder

With the createFolder function, you can create a new subfolder in one of the existing ones.

Creating a subfolder named "subfolder" in the root folder.

Delete a folder

Delete subfolders with the deleteFolder function.

Deleting a folder named "subfolder" from the root directory.

Folders, even if they are not empty, get deleted completely, including all of the content. They cannot be recovered afterwards!

Browse a directory

To show the content of a folder and all its subfolders, insert the path to the folder in the input field.

Displaying the contents of the root folder of the file connector.