Is there a way to avoid spending time in the two merge methods?

I noticed that the method: fit(ListDataSetIterator) spend a significant amount of time in DataSet.merge() / mergeFeatures and mergeLabels (I am doing it in batch). Does it mean at every call of fit method - it is rebuilding a DataSet for each batch? in my case - the content of my listDataSetIterator does not change so I guess this step is not necessary. Is there a way to avoid spending time in the two merge methods?