AI Assistants: A Framework for Semi-Automated Data Wrangling

Petříček, Tomáš; Burg, Gerrit J. J. van den; Nazabal, Alfredo; Ceritli, Taha; Jiménez-Ruiz, Ernesto; Williams, Christopher K. I.

doi:10.1109/tkde.2022.3222538

Cited by 5 publications

(3 citation statements)

References 52 publications

(65 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The second dataset is composed of the 256 CSV files in which CleverCSV was not able to accurately determine the dialect at the time of the research that originate the tool [15]. At the time of this research, 244 of these files were available online.…”

Section: Methodsmentioning

confidence: 99%

Detecting CSV File Dialects by Table Uniformity Measurement and Data Type Inference

García

2024

Preprint

View full text Add to dashboard Cite

The human-readable simplicity with which the CSV format was devised, together with the absence of a standard that strictly defines this format, has allowed the proliferation of several variants in the dialects with which these files are written. The latter has meant that the exchange of information between data management systems, or between countries and regions, requires human intervention during the data mining and cleansing process. This has led to the development of various computational tools that aim to accurately determine the dialects of CSV files, in order to avoid data loss during data loading by a given system. However, the dialect detection is a complex problem and current systems have limitations or make assumptions that need to be improved and/or extended. This paper proposes a method for determining CSV file dialects through table uniformity, a statistical approach based on table consistency and records dispersion measurement along with the detection of data type over each field. The new method has a 100% accuracy on a dataset with 148 CSV files composed of samples coming from a data load testing framework and some others added as verification of the parsing routines. In tests on truly messy data, the proposed solution outperforms the state-of-the-art tool by achieving an improvement of about 10% in the accuracy with which dialects are detected. Furthermore, the proposed method is accurate enough to determine dialects by reading only ten records, requiring more data to disambiguate those cases where the first records do not contain the necessary information to conclude with a dialect determination.

show abstract

Section: Methodsmentioning

confidence: 99%

Detecting CSV File Dialects by Table Uniformity Measurement and Data Type Inference

García

2024

Preprint

View full text Add to dashboard Cite

show abstract

“…The second dataset is composed of the 256 CSV files that CleverCSV could not accurately determine when conducting the research that led to the tool development [15]. At the time of this research, 244 of these files were available online.…”

Section: Methodsmentioning

confidence: 99%

Detecting CSV File Dialects by Table Uniformity Measurement and Data Type Inference

García

2024

Preprint

View full text Add to dashboard Cite

The human-readable simplicity with which the CSV format was devised, together with the absence of a standard that strictly defines this format, has allowed the proliferation of several variants in the dialects with which these files are written. The latter has meant that the exchange of information between data management systems, or between countries and regions, requires human intervention during the data mining and cleansing process. This has led to the development of various computational tools that aim to accurately determine the dialects of CSV files, in order to avoid data loss at data loading stage in a given system. However, the dialect detection is a complex problem and current systems have limitations or make assumptions that need to be improved and/or extended. This paper proposes a method for determining CSV file dialects through table uniformity, a statistical approach based on table consistency and records dispersion measurement along with the detection of data type over each field. The new method has a 100\% accuracy on a dataset with 148 CSV files composed of samples coming from a data load testing framework and some others added as verification of the parsing routines. In tests on truly messy data, the proposed solution outperforms the state-of-the-art tool by achieving an improvement of about 10\% in the accuracy with which dialects are detected. Furthermore, the proposed method is accurate enough to determine dialects by reading only ten records, requiring more data to disambiguate those cases where the first records do not contain the necessary information to conclude with a dialect determination.

show abstract

“…The solution presented in this research is compared head to head in the CSV dialect detection field with the alternative method CleverCSV [5], a tool showing an accuracy of close to 97% [15].…”

Section: Methodsmentioning

confidence: 99%

Detecting CSV File Dialects by Table Uniformity Measurement and Data Type Inference

García

2024

Preprint

View full text Add to dashboard Cite

The human-readable simplicity with which the CSV format was devised, together with the absence of a standard that strictly defines this format, has allowed the proliferation of several variants in the dialects with which these files are written. The latter has meant that the exchange of information between data management systems, or between countries and regions, requires human intervention during the data mining and cleansing process. This has led to the development of various computational tools that aim to accurately determine the dialects of CSV files, in order to avoid data loss during data loading by a given system. However, current systems have limitations and make assumptions that need to be improved and/or extended. This paper proposes a method for determining CSV file dialects through table uniformity, a statistical approach based on table consistency and records dispersion measurement along with the detection of data type over each field. The new method has a 100% accuracy over a dataset with 147 sampling CSV files from a benchmark framework. Furthermore, the proposed method is accurate enough to determine dialects by reading only ten records, requiring more data to disambiguate those cases where the first records do not contain the necessary information to conclude with a dialect determination.

show abstract

AI Assistants: A Framework for Semi-Automated Data Wrangling

Abstract: version of paper accepted for publication in IEEE Transactions on Knowledge and Data Engineering.

Cited by 5 publications

References 52 publications

Detecting CSV File Dialects by Table Uniformity Measurement and Data Type Inference

Detecting CSV File Dialects by Table Uniformity Measurement and Data Type Inference

Detecting CSV File Dialects by Table Uniformity Measurement and Data Type Inference

Detecting CSV File Dialects by Table Uniformity Measurement and Data Type Inference

Contact Info

Product

Resources

About