How to create my own task?

[GeorgeOrfanidis]

Hello everyone,

I am trying to use the finetune_classifier.py (https://github.com/dmlc/gluon-nlp/blob/v0.10.x/scripts/bert/finetune_classifier.py) to finetune my own defined task. I have created the training dataset, dev dataset, and testing dataset. However, looking at the command line arguments and specifically on the following:

 parser.add_argument(
        '--task_name',
        type=str,
        choices=['MRPC', 'QNLI', 'RTE', 'STS-B', 'CoLA',
                 'MNLI', 'WNLI', 'SST', 'XNLI', 'LCQMC', 'ChnSentiCorp'],
        help='The name of the task to fine-tune. Choices include MRPC, QQP, '
             'QNLI, RTE, STS-B, CoLA, MNLI, WNLI, SST.')

you can specify the above task names. My question is, how can I modify or use other gluon python files to create my own defined task using the training dataset, dev dataset, and testing dataset that I have created? And then use this finetune_classifier.py to finetune bert for the task that I am implementing.

Thank you in advance for any guidance and help.

[szha]

If your dataset happens to follow the same format as GLUE benchmarks, you can implement the GlueTask interface:

gluon-nlp/src/gluonnlp/data/classification.py

Line 29 in e65cd41

class GlueTask:

It includes a number of metadata fields that are relevant for a dataset.

Here's an example from Quora Question Pair dataset where each sample is a pair of sentences and a label.

gluon-nlp/src/gluonnlp/data/classification.py

Lines 139 to 180 in e65cd41

	class QQPTask(GlueTask):
	"""The Quora Question Pairs task on GlueBenchmark.
	Examples
	--------
	>>> QQP = QQPTask()
	>>> QQP.class_labels
	['0', '1']
	>>> type(QQP.metrics.get_metric(0))
	<class 'mxnet.metric.Accuracy'>
	>>> type(QQP.metrics.get_metric(1))
	<class 'mxnet.metric.F1'>
	>>> import warnings
	>>> with warnings.catch_warnings():
	... # Ignore warnings triggered by invalid entries in GlueQQP set
	... warnings.simplefilter("ignore")
	... QQP.dataset_train()[0]
	-etc-
	'train'
	>>> QQP.dataset_test()[0]
	-etc-
	'test'
	>>> len(QQP.dataset_test()[1])
	390965
	"""
	def __init__(self):
	is_pair = True
	class_labels = ['0', '1']
	metric = CompositeEvalMetric()
	metric.add(Accuracy())
	metric.add(F1(average='micro'))
	super(QQPTask, self).__init__(class_labels, metric, is_pair)
	def get_dataset(self, segment='train'):
	"""Get the corresponding dataset for QQP.
	Parameters
	----------
	segment : str, default 'train'
	Dataset segments. Options are 'train', 'dev', 'test'.
	"""
	return GlueQQP(segment=segment)

Note that it returns a GlueQQP dataset object but for your custom dataset you don't have to implement it this way.

gluon-nlp/src/gluonnlp/data/glue.py

Line 259 in e65cd41

class GlueQQP(_GlueDataset):

Dataset classes are simply list-like containers where it implements __getitem__ and __len__. You can simply use a list of samples for the same purpose. Alternatively, if your file is in TSV/CSV format, you can use the built-in TSVDataset class for loading it.

gluon-nlp/src/gluonnlp/data/dataset.py

Line 84 in e65cd41

class TSVDataset(SimpleDataset):

Hope it helps.